DREAM: Domain-free Reverse Engineering Attributes of Black-box Model
Authors: Rongqing Li
Jiaqi Yu
Changsheng Li
Wenhan Luo
Ye Yuan
Guoren Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Model Attribute inference
domain-free method

Abstract: Deep learning models are usually black boxes when deployed on machine learning platforms. Prior works have shown that the attributes (e.g., the number of convolutional layers) of a target black-box neural network can be exposed through a sequence of queries. There is a crucial limitation that these works assume the dataset used for training the target model to be known beforehand, and leverage this dataset for model attribute attack. However, it is difficult to access the training dataset of the target black-box model in reality. Therefore, whether the attributes of a target black-box model could be still revealed in this case is doubtful. In this paper, we investigate a new problem of Domain-free Reverse Engineering the Attributes of a black-box target Model, called DREAM, without requiring the model's training dataset available, and  put forward a general and principled framework by casting this problem as an out of distribution (OOD) generalization problem. At the heart of our framework, we devise a multi-discriminator generative adversarial network (MDGAN) to learn domain invariant features. Based on these features, we can learn a domain-free model to inversely infer the attributes of a target black-box model with unknown training data. This makes our method one of the kinds that can gracefully apply to an arbitrary domain for model attribute reverse engineering with good generalization ability.  Extensive experimental studies are conducted and the results validate the superiority of our proposed method over the baselines.

Under review as a conference paper at ICLR 2023
DREAM: DOMAIN-FREE REVERSE ENGINEERING AT- TRIBUTES OF BLACK-BOX MODEL
Anonymous authors Paper under double-bling review
ABSTRACT
Deep learning models are usually black boxes when deployed on machine learning platforms. Prior works have shown that the attributes (e.g.. the number of con- volutional layers) of target black-box neural network can be exposed through a sequence of queries. There is a crucial limitation that these works assume the dataset used for training the target model to be known beforehand, and leverage this dataset for model attribute attack. However, it is difficult to access the training dataset of the target black-box model in reality. Therefore, whether the attributes of a target black- box model could be still revealed in this case is doubtful. In this paper, investigate new problem of Domain-free frre Reverse Engineering the Attributes of a black-box target Model, called DREAM, without requiring the availability of target model's I's training dataset, and put forward general and principled framework by casting this problem as an out of distribution (OOD) generalization problem. At the heart of our framework, we devise a disriiidisrr generative adversarial network (MDGAN) to learn domain invariant features. Based on these features, we can learn a domain-fres model to inversely infer the attributes of a target black-box model with unknown training data. This makes our method one of the kinds that can gracefully apply to an arbitrary domain for model attribute reverse engineering with strong generalization ability. Extensive experimental studies are conducted and the results validate the superiority of our proposed method over the baselines.
INTRODUCTION
With its commercializatioo, machine learning as a service (MLaaS) is becoming more and more popular, and providers are paying more attention to the privacy of models and the protection of intellectual property. Generally speaking, the machine learning service deployed on the cloud platform is a black box, where users can only obtain outputs by providing inputs to the model. The attributes of the model such as architecture, training set, training method, are concealed by provider. However, if such deployment is safe? Once the attributes of the model are revealed, it will be beneficial to many downstream attacking tasks, e.g., adversarial example generation (Moosavi-Dezlte et al.) 2016), model inversion (He et al. 2019). etc.
(Oh et al. 2018) has conducted model reverse engineering to reveal model attributes, as shown in the left of Figure | They first collect large set of white-box models which are trained based on the same datasets as the target black-box model, e.g., the MNIST hand-written dataset (Lecun et al. 1998). Given a sequence of input queries, the outputs of white-box models can be obtained. After that, meta-classifier is trained to learn mapping between model outputs and model attributes. For inference, outputs of the target black-box model are fed into the meta-classifier to predict model attributes. The promising results demonstrate the feasibility of model reverse engineering.
However, crucial limitation in (Oh et al., 2018) is that they assume the dataset used for training the target model to be known in advance, and leverage this dataset for meta-classifier learning. In most application cases, the training data of a target black-box model is unknown. When the domain of training data of the target black-box model is inconsistent with that of the set of constructed white-box models, the meta-classifier is usually unable to generalize well on the target black-bor box three
model. To verify this point, we train three black-box models with the same architecture different datasets, Photo, Cartoon and Sketch(Li et al. (017),, respectively. We use the method in (Oh et al. 2018) train meta-classifier the white-box models which are trained on the Cartoon

---

Under review as a conference paper at ICLR 2023
Previous
Our DREAM framework
Training
loutputs, labels)
Training
(Multi-domaiin outputs, labels)
MNISTT
White bexes
Meta
Other digit datasets White baves
train
train
train
MDGAN
query
Attributes
query
Attributes
Inference
outputs
Inference
outputs
MNNIST
Unknown dataset train
Black box
train
Trained
MDGAN
Figure 1: Previous work (left) assumes the dataset used to train the target black-box model is given beforehand, and requires to use the same dataset to train white-box models. Our DREAM framework (right) relaxes the condition that training data of black-box model is longer required to be available, and proposes a domain-free method to infer attributes of black-box model. Our idea is casting the problem into an out-of-distribtiion learning problem, and designing GAN (Goodfellow al. 2014) based network (MDGAN) to learn domain invariant features for black-box model attribute inference.
dataset. After that, we use the trained meta-classifier to infer attributes of three black-bo -oxx models, respectively. As shown in Figure 2 when the training dataset of black-box models and white- box models are the same (i.e.. Cartoon), the performance reaches about 80%, otherwise, it sharply drops to about 40%, close to random guess. The huge gap shows that it is not trivial investigate model reverse engineering with the assumption of the training dataset of black-box model not available. Furthermore, the training set for black-box model changes, (Oh et al.] 2018) needs to retrain the whole set of white-box models to obtain promising result, which extremely time-consuming.
In this paper, we investigate the problem black-box model attribute reverse engineering, longer requiring training data of the target model is available, as shown the right of Figure [] Obviously, when feeding the input queries to models with the same architecture but training data of different domains, the output distributions of these models are usually different. Thus, in our problem setting, key point is how bridge the gap between the
Average Accurncy
60
50
60
30
20
output distributions of white-box and target black-box models, due to the lack of the target model's training data. An ideal meta-classifier should be well trained based outputs of white-box models, and predict well on outputs of the target black-bo box model, even if white-bor box and black- box models are trained using data of different domains.
10
Randox
Sketah
Photo
Cartoon
Training set of of box
Figure 2: The performance of (Oh et al. 2018 on three datasets.
In light of this, propose a novel framework DREAM: Domain-free Reverse Engineering the Attributes black-box Model. In the field of computer vision, out of distribution generalization learning has been widely studied in recent years (Shen et al., 2021), where its main goal is to learn a model on data of one multiple domains, and generalize well on data of another domain unseen during training. One kind of mainstream OOD learning approaches is to extract domain invariant features from data of multiple different domains, and utilize the domain invariant features for downstream tasks (Li et al.) 2018, Kim et al. 2021] Zhou et al.) 2021b). These methods mainly focus on image or video data, and have shown powerful performance. Back to our problem, the black-box models deployed on cloud platform provide their functionality and which categories they can output. Therefore, we can collect data with the same label but different distribution as domains to train white-box models and obtain their probability outputs. since the data concentrate is related to the outputs of machine learning models, c.g., probability value, how to design an effective OOD learning method over this type of data has not been explored. To this end, we design multi-diseriminatoo generative adversarial network (MDGAN) to learn domain invariant features from the outputs of white-box models trained multi-domain data. Based learnt domain invariant features, learn domain-free reverse
such problem
out of distribution (OOD) generalization problem, and
2

---

Under review as a conference paper at ICLR 2023
model which can well infer the attributes of target black-box model trained using data of arbitrary domain.
Our contributions summarized follows: 1) We provide the first study on the problem of domain- free reverse engincering the attributes of black-box models, and cast it as an out of distribution (OOD) generalization problem; 2) We propose generalized framework, DREAM, which can address the problem of inferring the attributes of a black-box model with an arbitrary training domain; 3) We constitute the first attempt to explore learning domain invariant features from probability representations, in contrast to traditional image representations; 4) We perform extensive experiments and analyze the results, demonstrating the effectiveness of our method.
2 RELATED WORK
Reverse Engineering of Model Attribute. Its goal is to reveal attribute values of a target model, such as model structure, optimization method, hyperparameters, elc. Current research efforts focus on two aspects, hardware (Yan et al., 2020 Hua et al. 2018) and software (Oh et al.) 2018 Wang & 2019). The hardware-based methods utilize information leaks from side-channel (Hua et al. Yan et al. 2020) or unenerypted PCIe buses (Zhu et al., 2021) to invert the structure of deep seftware-based methods reveal model attributes by machine learning. (Wang & Gong 2019) steals the trade-off weight of loss function and the regularization term. They derive over-determined linear equations and solve the hyperparameters by least-square method. KENNEN (Oh et al. 2018) prepares a set of white-box models, and then trains a meta-classifier to build a mapping between model outputs and their attributes. It is the most related work to ours. However, a significant difference is that, Oh et al. 2018) requires the data used to train the target black- box model to be given beforehand, while our method relaxes this condition, i.c., no longer require the training data of target model to be available. Thus, we attempt to solve a more practical problem.
Gong
2018
neura networks.
Model Functionality Extraction. It aims to train clone model that has similar model functionality to that of the target model. To achieve this goal, many works have been proposed in recent years (Orekondy al., 2019 Truong et al., 2021 Papernot al., 2017). (Orekondy et al.) 2019) uses an alternative dataset collected from Internet to query the target model. (Papernot et al. 2017) assumes part of dataset is known, and then presents dataset augmentation method to construct the dataset for querying the target model. Moreover, data-free extraction methods (Kanyyapaa et al. 202T Truong al. (022]) query a target model through data generated by a generator, and use zero-order gradient approximation to approximate the gradient of the target model. Different from the methods mentioned above, our goal is to infer the attributes of a black-box model, rather than stealing the model function.
Membership Inference. Its goal is to determine whether sample belongs to the training set of a model (He et al., [2020] Choquette- Caequeette-hhooooeo et al.) [202T] Rezaei & Liu, 2021]. Although inferring model attribute is different from the task of membership inference, the technique in Oh et al. (2018) is actually similar to those of membership inference attack. However, as stated aforementioned. when the domain of training data of the target black-box model is inconsistent with that of the set of white-box models, the method is usually unable to generalize well because of the OOD problem.
OOD Generalization. The goal of OOD Generalization is to deal with the inevitable shifts from a training distribution to an unknown testing distribution (Shen et al.
2021 Existing methods
mainly fall into three categories: domain generalization
et al. 2021
Li al. 2018 Zhou et al.
2021b a Hu et al., (020). causal learning (Arjovsky et | 2019 Creager et al.] 2021 1 Krueger et al.
2021 Mahajan et al. (021)) and stable learning (Shen et al. 2020 Kuang et al. [2200] [hhang et al.
2021 Kuang et al. 2018). Domain generalization attempts to learn invariant representations among different domains. The work closest to us is ADA (Ganin et al. 2016) which uses an adversarial strategy between a feature extractor and a discriminator to learn domain invariant features. ADA designed for domain adaptation task (only two domain). However, we aim to solve a domain generalization problem that handle more than two domains, single discriminator cannot learn domain invariant feature between multiple domains. Causal learning and stable learning aim to search for causal features to ground-truth labels from data and filter out label-unrelated features. The former makes existing causal features invariant, while the later focuses on the effective features strongly related to labels by reweighting attentions. The above methods mainly focus image video. How to design an effective OOD learning method for attribute inference of black-bor model has not been ex explored so far.
3

---

Under review as a conference paper at ICLR 2023
pretrain
Embedding
rcal
Pale
fake
AUE dighiin
domalall
Real
unipati
prob
madeh
Fake Ravens Muded
fakic dreniald3
Real
Oyte
mutts-domain
Skrihh
Ma
Diveriitaaer
Fale
query
Rasd
Figure 3: An illustration our DREAM framework. 1) In the left part, we prepare large number of box models from different domains. In each domain, models have various combinations of attributes, then we input model output. 2) In the right part, the core idea is designing a diiiiie-ddiriimitttoo GAN network (MDGAN) to learn domain invariant features from the outputs of white-box models trained on multi-domain data. After that, a domain-free reverse model is learnt based on domain invariant embeddings, and is used to infer the attributes of black-box model with arbitrary domains.
queries into each white-box model obtain a multi-domain
3 PROPOSED METHOD
3.1 PRELIMINARIES
KENNEN (Oh et al. 2018). Given black-box model B. model attribute reverse engineering in (Oh et al. 2018) aims to build meta-classifier + : B - A, where the set of model attributes including model architecture, optimizer and training hyperparameters, etc. Concretely, they firstly collect large set of white-box models F containing different attributes combination, and train these white-box model based on the same training data D as that of the target black-box model. Then outputs O is obtained by querying these white-box models with a sequence of input images. Finally they build mapping & from outputs O to model attributes A, realized as meta-classifier D. At the inference phase, the meta-classifier takes outputs from target model as input and predicts the corresponding attributes.
Problem Formulation. As aforementioned, there is a strict constraint in (Oh et al. 2018 that they assume the training dataset D of the target model to be given in advance, and leverage D for learning meta-classifier Q. In most scenarios, especially on public machine learning platforms, it is difficult to access the training data of target black- -ox model, which significantly limits the applications of (Oh et al., 2018). To mitigate this problem, we provide a new problem setting by relaxing the above constraint, i.c., we no longer require the training data D of the target black-box model to be available. Thus, our goal is to learn a domain-free reverse classifier & that is trained based on outputs of white-box models F, and predict well for the target black-box model, even white-box and black-box models are built based on training data of different domains.
3.2 DREAM FRAMEWORK
To perform domain-free free black-box model attribute reverse enginecring, cast this problem into an out-of-distribute (OOD) generalization learning problem, and propose a novel framework DREAM, shown in Figure 3 Our DREAM framework consists of two parts: In the left part of Figure 3 train a number of white-box models with training sets from different domains. Models of each domain enumerated with various model attributes. All of these models constitute a model covering different domains (please refer to Sect. 4. for more details). Next, we prepare queries as input to these models. For each domain, we sample an equal number of images from the corresponding dataset, and concatenate them as a batch of queries. These queries are sent to each model, and outputs of the model are fed into the other module of our DREAM framework, as shown in the right part of Figure 3 The core idea is to design multi- dilriddiseriminator generative adversarial network (MDGAN) to learn domain invariant features, where MDGAN consists of multiple discriminators corresponding to different domains and one generator across multiple domains. The generator aims to learn domain invariant features, and each discriminator intends to make the learnt feature distributions other domains to fit that of the domain itself. In this way, the generator is capable of learning domain invariant features. Based on the learnt domain invariant features, we can learn a domain- free reverse model infer the attributes of black-box model with arbitrary domain.
4

---

Under review as a conference paper at ICLR 2023
3.3 MULTI-DOMAIN OUTPUT PREPARATION
The multi-domain output can be taken as a representation of a white-box model, and is fed into MDGAN to learn domain invariant features. Specifically, from the dataset of each domain to obtain a query set Q = {u, ) where N the number of queries. We denote training model set from each domain as F= [f f2 ..... fr), where f consists of K models of tth domain. Then, we input each query E Q into models f of 7th domain to get an output O E RKxC, where Oj represents K outputs of gdh domain for a query. We obtain O E R xxXNN by concatenating N outputs. Finally, we derive multi-domain outputs as O 00 ...., O"] E I
sample an equal number of images
The core idea of MDGAN is to learn embeddings for each domain by parameter sharing generator, and make the distributions of different domains as close as possible by multiple discriminators.
3.4 MULTT-DISCRIMINATON GAN (MDGAN)
After preparing multi- domain outputs, devised GAN based network, MDGAN, to learn domain invariant features from the outputs of white-box models trained on multi-domain representation. To better present, we take Figure 4o illustrate the idea behind MDGAN. Assume there are two kinds of inputs, O and 02, from two domains. When feed them into the generator G, we can obtain the corresponding embeddings 2! and 2?, respectively. After that, we feed z' and x² to the discriminator D1, where D is expected to output "real" label for z! and output "fake" label for 22. By jointly training G and D2 based on minmax optimization, the distribution of x² is expected meantime, we also feed 2 and 2² to the discriminator D2. Differently, D2 is expected to output a "real" label for 2² and output "fake" label for 2 By jointly training G and D2, the distribution of 2² is expected to move towards that of 22, In this way, zt and z² generated by the generator G become domain invariant representations.
D2
-ncal fake
D2 real or fake
Figure 4: An example to illustrate the idea behind MDGAN.
towards that of 2 In the
Formally, we define G)); : 0-2. 2. The generator G sharing with parameter 0g across domains maps multi-domain outputs O into the latent feature z. We also define m discriminators (D' )- Each discriminator D'(2): 2 > [0, 1] outputs a scalar representing the probability that = comes from the jth domain rather than others. For D' we treat the correct label of an embedding in the ith domain, i.e., O², True, while others as False. Then we divide outputs into two groups, {Or+ and {0'p which are defined as:
(0t) = (O'); (0%) = (0'1) + i); UiOHJ =0
J+1 The training goal of Dd is maximize the probability of assigning the correct label to features both from the ,th domain and other domains, while the generator G is trained against the discriminator to minimize log(1 (x)))) In other words, it is min-max game between the ith discriminator D and generator G with value function V, formulated as:
EEZ-(oS) Di(G())) (2) H+t During optimizing the min- max adversarial loss for G and D', the distributions of model outputs from the thh domain and other domains become closer. After and all D are well trained, G will embed multi-domaii model outputs into an invariant feature space, where each discriminator cannot figure out which domain the outputs of white-box box models are from.
min max V(D', D
(og
3.5 DOMAIN-FREE REVERSE MODEL
Then, the domain-free reverse classifier classify the domain invariant features produced by the generator. We denote features z produced by G(O;6,) as
z = (G0"" p mmx
5

---

Under review as a conference paper at ICLR 2023
Where d' is the number of feature dimensions. We define the domain-free reverse classifier as (22; 0c) parameterized by c.. We obtain probability p(z") ) for each possible model attribute as:
(c)()(22))
p(=²) softmar(((2)) (x
E (
The target is to minimize the cross entropy between the predicted p(=') and ground-trute of model attribute values y:
min E C(O)
Σ v'log(p(z"))
min E G0OO [-y² Tog(p(z))]
At inference phase, given the same queries as the white-box model, the outputs of black-box model from an unknown domain are fed into the generator G, and then the output of G' is fed into the reverse classifier Q achieving domain-free prediction black-box model attributes.
3.6 OVERALL MODEL AND TRAINING STRATEGY
After introducing all the components, give the final loss function based Eq.
2 and 5 as:
min G Di,ssism
V(D*, G) (o-(0))) [log [logD((G<x)) + E E Ben(o)) [log(1 D+(G(=))))
+XE [-yg)og(p(=))]-
where A is a trade-off parameter. We observe if we firstly train MDGAN, and then optimize domain- free reverse classifier, the generator of MDGAN will converge to trivial solution. In other words, tends to produce identical features, resulting in a reverse model which cannot predict model attributes correctly. Thus, we design a training strategy: we first optimize all discriminators D' and then jointly optimize the generator and the domain-free reverse classifier. We repeat the above processes, until the algorithm converges. The proposed training strategy represented in Algorithm I of Appendix A.3.
EXPERIMENTS
4.1
DATASET CONSTRUCTION
Following (Oh et al. 2018), we train number of models which are constructed by enumerating all possible attribute values. The details of the attributes and their values are shown in Table The number of models with all possible combinations of the attributes is 5, 184. We also initialize each model with random seeds from and 999, yielding 5, 184,0000 unique white-box models. For each domain, randomly sample and train 10,000 white-box models from 5,184,000 models. Then alse sample 5000, 1000, 1000 from 10,000 white-box box models as the training set, validation set, and testing set. Next, we introduce the details of our datasets.
PACS-modelset. PACS an image dataset that has been widely used for OOD learning (Li al.) 2017). In this experiment, we use for evalu- ating our domain-free black-box model attribute inference framework DREAM. We utilize three domains, including Photo (1,670 images), Car- toon (2,344 images) and Sketch (3,929 images), construct our dataset. In our dataset, each do- main contains 7 categories. For each domain we train 10,000 models and we combine them as PACS-modelset (30,000 models in total).
Table 1: Attributes and the corresponding values.
Attribute
Values
#Activation
ReLU, PReLU, ELU, Tanh
#Dropout
Yes, No
#Max pooling
Yes, No
#Batchnorm
Yes, No
#Krnell size
3,5
#Cony layers
2, 3,
#FC layers
2, 4
MEDU-modelset. MEDU is a set of written digit recognition dataset, with 4 domains col- lected from MNIST (Lecun et al. 1998), USPS (Hull 1994). DIDA (Kusetogulaan EMNIST (Cohen et al. (2017). Each domain contains different styles of hand written digit from 0 to 9. We train 40,000 models MEDU-modelset and each domain contains 10,000 models.
#Optimizer
SGD, ADAM, RMSprop
#Batch size
32, 64, 128
al.) and

---

Under review as a conference paper at ICLR 2023
Table Model attribute classification accuracy (%) on PACS- Red and blue indicate the best and second best performance, respectively.
Attributes #conv
Method
Avg
#act
#drop
#pool
#ks
#Ic
#opt
#bs
#bn
Random
25.00
50.00
50.00
50.00
33.33
33.33
33.33
33.33
50.00
39.81
SVM
37.80
50.30
54.80
53.60
34.00
36.60
37.00
45.70
58.80
45.40
KENNEN
39.07
50.68
59.42
61.31
36.18
39.33
37.88
44.16
59.74
47.53
SelfReg
25.58
52.26
54.98
50.18
34.12
35.25
34.61
33.78
50.76
41.28
MixStyle
39.63
53.23
61.83
59.44
35.66
38.75
37.89
43.75
57.09
47.47
P
MMD
38.88
54.70
60.46
56.54
35.38
36.66
35.66
40,50
61.04
46.65
DREAM
43.84
59.19
66.09
64.24
39.59
42.04
40.49
47.83
68.12
52.38
SVM
25.80
49.20
50.70
55.80
37.20
38.10
30.80
42.30
65.30
43,91
KENNEN
32.99
52.50
54.23
56.57
37.19
40.53
33.47
37.17
68.39
45.89
SelfReg
25.97
51.42
56.20
50.03
35.04
35.52
36.09
35.58
56.17
42.44
MixStyle
32.10
50.76
55.44
54.18
36.18
37.87
34.65
38.69
60.26
44.46
C
MMD
29.56
53.02
54.70
53.82
35.38
36.36
35.98
37.24
57.58
43.75
DREAM
37.53
55.89
61.18
57.32
38.58
39.60
38.32
45.01
65.16
48.73
SVM
23.80
47.60
47.40
45,80
33.80
34.50
31.80
34.30
53.10
39.12
KENNEN
34.64
50.10
53.07
52.01
34.61
37.11
35.78
37.04
55.27
43.29
SelfReg
27.07
54.32
51.39
53.07
36.99
36.82
35.47
34.17
61.80
43,46
MixStyle
37.78
51.71
54.16
53.60
34.53
36.16
36.36
36.02
59.42
44.42
S
MMD
31.96
52.94
56.84
52.78
38.18
38.20
36.20
35.92
57.56
44.51
DREAM 42.24
55.68
61.82
58.34
39.55
38.39
38.51
41.39
74.39
50.03
In the experiment, we set the number of queries N 100, We use Adam (Kingma & Ba 2014) as the optimizer, where the learning rate is set to 10-5 for the generator and discriminators. and the learning rate 3 is set to 10 -4 for the reverse model. The batch size b set as 100. The trade-off parameter À is tuned from {0.001, 0.01, 0.1, 10) based the validation set. Parameter sensitive analysis can be found in Appendix. In addition, the generator and discriminators are implemented two-layer MIP, respectively. where ReLU is used the non-linear activation function. All experiments are conducted on 4 NVIDIA RTX 3090 GPUs, PyTorch 1.1 1.0 platform.
We compare our DREAM with 6 baselines including Random choice, SVM, KENNEN (Oh al. 2018), SelfReg (Kim et al. 2021), MixStyle (Zhou et al. 2021b), MMD (Li et al. 2018). To compare fairly, we select a variant KENNEN (denoted as KENNEN taking fixed queries as input. which is the same ours. Moreover, we also take three typical OOD generalization methods, SelfReg, MixStyle and MMD, as baselines to verify the effectiveness of our proposed MDGAN network for learning domain invariant features. SelfReg aims to draw samples of similar categories between all domains closer and samples of different categories farther; MixStyle captures style information of images by the CNN layer, and it performs style mixing at the layer; MMD adopts maximum mean discrepancy loss between two domains. To apply OOD baselines, we first take probabilities as input to learn invariant features by them, and then adopt MLP on these features to predict model attributes. In addition, we take SVM as a basic baseline without considering different domain outputs.
We adopt the "leave-one-dee ceaeeeeoneeommmaae domain-out' scheme to split the source and target domains. For each dataset, we in turn take one domain as the target domain and the rest domains as source domains. We run the experiment 10 trials and report the average accuracy each split.
4.2 EXPERIMENTAL. RESULTS AND ANALYSIS
Overall Performance. Table 2 and report the overall performance of different methods on the respectively. The left-most column in each table indicates the target domain (the rest ones are source domains). The performance achieved by our proposed DREAM is better than that of all baselines terms of the average result models attributes. For individual attribute, our method outperforms other methods in most of the cases. Our method is better than KENNEN, which illustrates our method benefits from learning domain invariant features and learning domain-free reverse model. Moreover, the three OOD learning methods, which indicates it is necessary to design new methods to extract domain invariant features for model attribute inference of black-box models.
M-CS-modelset and
method achieves better performance than
7

---

Under review as a conference paper at ICLR 2023
Table Model attribute classification accuracy (%) on MEDU-modelset. Red and blue indicate the best and second best performance, respectively.
Attributes
Method
Avg
#act
#drop
#pool
#ks
#conv
#Ic
#opt
#bs
#bn
Random
25.00
50.00
50.00
50.00
33.33
33.33
33.33
33.33
50.00
39.81
SVM
45.60
49.40
62,90
59.20
38.80
40.10
35.50
35.00
75.30
49.09
KENNEN
51.18
50.67
62.99
57.36
38.32
35.84
41.57
35.75
77.87
50.17
SelfReg
28.00
53.57
53.43
50.78
35.97
36.39
35.98
36.23
53.96
42.70
MixStyle
50.27
51.72
62.66
57.32
37.88
36.34
43.11
38.00
82.61
51.10
M
MMD
44.57
59.67
66.37
57.27
39.63
37.27
42.10
37.60
81.37
51.76
DREAM
51.01
62.32
64.28
58.39
40.96
38.11
45.37
38.96
81.99
53.49
SVM
40.00
48.70
69.20
51.60
40.20
36.90
35.80
30.
79.90
48 48.04
KENNEN*
45.66
51.01
65.26
53.25
40.28
36.35
41.96
36.16
81.30
50.14
SelfReg
27.29
52.83
53.32
52.85
33.68
35.05
35.26
35.32
53.74
42.15
MixStyle
43.68
51.35
67.87
57.15
42.50
39.30
42.10
38.79
82.46
51.69
MMD
42.03
58,43
66.27
60.80
40.80
38.67
40.00
39.97
84.00
52.33
DREAM
45.55
64.98
74.16
60.71
44.45
42.45
47.37
41.03
91.00
56.86
SVM
45.00
47.80
54.60
45.50
29.40
37.60
43.30
36.50
63.70
44.82
KENNEN
42.73
52.06
55.27
52.02
34.89
38.90
38.98
36.27
54.97
45.12
Selfeeg
26.31
54.29
53.23
52.33
34.96
35.72
36.49
35.39
59.11
43.09
MixStyle
45.26
52.32
55.91
51.39
34.22
38.70
38.31
38.03
57.44
45.73
MMD
39.00
59.20
59.63
55.93
35.93
38.33
37.93
37.50
54.40
46.43
DREAM
49.63
64.50
59.30
57.13
39.52
44.59
42.09
40.19
59.68
50.74
SVM
43.40
50.50
47.60
52.50
30.30
32.30
41.00
36.60
49.40
42.62
KENNEN
43.38
50.88
51.41
53.19
36.35
35.59
36.66
34.56
55.62
44.18
SelfReg
26.81
52.16
55.46
52.47
36.18
36.43
36.53
35.90
55.34
43.03
Mix
41.05
53.80
50.49
52.93
35.26
33.68
36.92
34.75
59.34
44.25
MMD
39.33
55.87
52.67
53.23
39.20
34.33
35.90
36.90
60.73
45.35
DREAM
42.34
58.72
58.58
54.41
37.90
37.81
40.42
38.36
63.39
47.99
What is more, observe that DREAM cannot outperform other baselines in some cases. The reasons might be: 1) DREAM vs. OOD learning baselines. As we have mentioned, the OOD learning methods aim learn domain invariant space from different domains. Once features of different domains are excessively pulled close, the classification accuracy would be influenced. Thus, the trade-off between invariant space learning and classification learning is vital for the performance of reverse engineering. Moreover, the trade-off for each attribute is not identical. Taking MMD in Table E as an example, the best trade-off hyperparameter conduces to predict attribute #ks better than other attributes. Similarly, the best trade-off hyperparameter of DREAM conduces to better predict attributes except for #ks. 2) DREAM vs. KENNEN and SVM. Our proposed DREAM has stronger ability to fit complicated data, while SVM and KENNEN (only shallow MLP) are weaker in the scenery of complicated data. In the scenery of casier cases, e.g., #act, #ks, #fc in of MEDU (shown in Table 3) DREAM is more likely overfit due more parameters, degrading the performance of our method. However, our method generally performs better than SVM and KENNEN most cases.
Visualization of Generated Feature Space. To further verify the effectiveness of proposed method, we utilize I-SNE (Van der Maaten & Hinton 2008) to visualize samples in the domain invariant feature space learnt by the generator G in MDGAN. The visualization is carried out on PACS-modelset. We take C (cartoon) and P (photo) as source domains to train white-box models, and use S (sketch) the unseen target domain train black-box model. As shown Figure samples from the three different domains are grouped into individual clusters at the 1st epoch. This illustrates their distributions are indeed different in the beginning. Distributions of source domains (C and P) become closer from epoch to 5. Then, our method embeds features from the unseen domain (S) and the samples from the target domain also become closer to the source domains at the 10th epoch, indicating that our generator is able to generalize an unseen domain into the feature space where the source domains are in. Finally, both source and target domains are transformed into an invariant feature space. For MMD and Mixstyle in 5 and Figure 5jc). the distributions of features does not become closer as the training proceeds. For SelfReg in Figure the features are pulled closer to some extent from epoch #1 epoch #10, and part of samples from the unseen target
8

---

Under review as a conference paper at ICLR 2023
Epoch #1
Epoch #5
Epoch #10
Epoch #10
10.
13.
Ainioo leat
20
30
aas 2011
10
10-
eakulth
-5
-10
-20
-10
10
-10
-10
10
-20
Epoch
Epach
Epoch
Epoch
20
<shats
<choto
15
<Dhot:
15-
<2hol:
-16-
-15
-30
-5
5
13
-20-15-0
30
15 15
-15-15
15
22
Epoch #1
Epoch #5
Epoch #10
Epoch #100
15
saritin
10
10
-1b
-15
-15
Epoch
Epoch
Epoch
ccarolii sowect ner
19-
cooer
d)
-10
15
10075
Figure 5: T-SNE visualization of features of different domains produced by a) DREAM, b) MMD, C) MisStyle and SelfReg on PACS-modekee
0.925
0.525
Tokal Loss
0.90
Total Loss
Total Locs
0
0,900
0.875
6 6.875
0.85
0.850
0.850
0.825
0.825
0.80
200
0.800
50
150 Epoch
200
30
150
200
250
Epoch
Epoch
Figure 6: Convergence analysis. Loss curves of meta-classifier (Eq. 5) in the training phase on the three splits C, and (from left right) demonstrate the convergence of method.
domain S indeed become closer to the source domains at the 100 epoch. However, these feature distributions are not sufficiently tight.
We study the convergence of our algorithm the PACS-modelsee The curves of the meta-classifier's 's loss in the training phase are shown in Figure For all the three splits of domains (left to right), the loss decreases as the training proceeds and finally levels off.
Convergence Analysis
5 CONCLUSION
In this paper, we studied the problem of domain-free reverse engineering towards the attributes of black-box model with unknown domain data, and cast it as an OOD generalization problem. We proposed a new framework, DREAM, which can predict the attributes of a black- box model with an arbitrary training domain, and devised new GAN based network to leam domain invariant features in the scenario of attribute inference of black-box mode. Extensive experimental results demonstrated the effectiveness of our method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: (557::9907.02888 02893, 2019.
Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. International conference on machine learning, pp. 1964-1974. PMLR, 2021.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of
MNIST to handwritten letters. arXiv: 00000002033333 [cs], February 2017. URL http::/arxiii
htg/abs/1702. 05373
arXiv: 1702.05373 version: 1.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant
learning. International Conference on Machine Learning, pp. 2189-2200. PMLR, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):209-20330, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Yang He, Shadi Rahimian, Bernt Schiele, and Mario Fritz. Segmentations-leak: Membership inference attacks and defenses in semantic image segmentation. European Conference on Computer Vision pp. 519-535. Springer. 2020.
Zecheng He, Tianwei Zhang, and Ruby B Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 148-162, 2019.
Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain Generalization via Multidomain Discriminant Analysis. In Proceedings The 35th Uncertaimty in Artificial Intelligence Conference, pp. 292-302 PMLR, August 2020. URL httppss//pxoceecdnng..tttt .mr.. wtw.s//ewi..............e//s.. pr15s/v115/ ISSN: 2640-3498.
hu20a. html
Weizhe Hua, Zhiru Zhang, and G. Edward Suh. Reverse Engineering Convolutional Neural Networks Through Side-channel Information Leaks. In 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC), pp. 1-6, June 2018. doi: 10. (..110///C.2218.846555
J.J. Hull. A database for handwritten text recognition research. IEEE Transactions Pattern Analysis and Machine Intelligence, 16(5):550-554, May 1994. ISSN 1939-3539. doi: 10. (0.1109/34.291444 Conference Name: IEEE Transactions Pattern Analysis and Machine Intelligence.
Sanjay Kariyappa, Atul Prakash, and Moinuddin K Qureshi. MAZE: Data-Free Model Stealing Attack Using Zeroth- Order Gradient Estimation. 202/ IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13809-13818, Nashville. TN, USA, June 2021. IEEE. ISBN 978- 978-1-66544-509-2. doi: 10. 0009//CVRR4437.2222...00 leee /tteeeorrg/dccmmmnt/555777
URL [ttt:s::/
Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 1619-9628, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:141.66980 2014.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
Kun Kuang, Peng Cui, Susan Athey, Ruoxuan Xiong, and Bo Li. Stable prediction across unknown environments. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1617-1626, 2018
10

---

Under review as a conference paper at ICLR 2023
Kun Kuang, Ruoxuan Xiong, Peng Cui, Susan Athey, and Bo Li. Stable prediction with model misspecification and agnostic distribution shift. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4485 4492,4492244 2020.
Huseyin Kusetogullari, Amir Yavariabdi. Johan Hall, and Niklas Lavesson, Dida: The largest his- torical handwritten digit dataset with 250k digits. DIDA/ Accessed: 2021 2021-06-13.
//tp://gghthbb.t/// com/ madataset)
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998. doi: 10.1109/5.72699.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain Generalization With
Adversarial Feature Learning. pp. 5400-5409, 2018. URL httpt:tttt///ottttt// //tpaaccss///ecccsssss thecvf
Com/content__oo 2018/htmI/LL DoniainnmmL_DDomannnnn. http:///aa.....aoaaoneeeeaaaaaaa wwt.reeeeattiiiiiio With_CVPR
2018 pwwpp.pppp...
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, pp. 7313-7324. PMLR, 2021.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method fool deep neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2574-2582. 2016.
Seong Joon Oh, Max Augustin, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural networks. In 6th International Conference on Learning Representations, 2018.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff Nets: Stealing Functionality of Black-Box Models. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4949-4958, Long Beach, CA, USA, June 2019, IEEE. ISBN 978-1-7288 13-2933-8 URL htttps://eeexpporrtt hot.:ppoorr........ooo//cccceett
doi: 10. 10.1009/CVPR..21990000 00509.
8953839/
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical Black-Box Attacks against Machine Learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506-519, Abu Dhabi United Arab Emirates, April 2017. ACM. ISBN 978- 4-01----553-4444444 doi: 10. 00.11445/05297333533000 URL http:::/p::d acm. h:tg//ol/00..1155555555533 00044///33322733333330003 3053009
Shahbaz Rezaci and Xin Liu. On the difficulty of membership inference attacks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7892-7900, 2021.
Zheyan Shen, Peng Cui, Tong Zhang, and Kun Kunag. Stable learning via sample reweighting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5692-5699, 2020.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: survey, arXiv preprint arXiv:2108. 136244 2021.
Jean-Baptiste Truong, Pratyush Maini, Robert J. Walls, and Nicolas Papernot. Data-Free Model
Extraction. arXiv:201 1.14777 [cs], March 2021. URL http:://axiiii http:///axivv../////../ wwtp://rrgaaaa/aa/aaaoo 14779 arXiv: 2011.14779.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Binghui Wang and Neil Zhenqiang Gong. Stealing Hyperparameters in Machine Learning.
[cs, stat), September 2019. URL httt:://arxis wwtpswii.vrr/...o. /tbb//8022
05351 arXiv: 1802.05351.
11

---

Under review as a conference paper at ICLR 2023
Mengjia Yan, Christopher W. Fletcher, and Josep Torrellas. Cache Telepathy: Leveraging Shared Resource Attacks to Learn [DNN] Architectures. pp. 2003-2020, 2020. ISBN 978-1- 939133-17-5. URL https: www uwwnix. htt.e//w.......oooo...eeeeeeeeea presentati on on/yan yan
Xingxuan Zhang, Peng Cui, Renzhe Xu, Linjun Zhou, Yue He, and Zheyan Shen. Deep stable learning for outtr--distribution generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5372-5382, 2021.
Fan Zhou, Zhuqing Jiang, Changjian Shui, Boyu Wang, and Brahim Chaib-draa. Domain Gen- eralization via Optimal Transport with Metric Similarity Learning. Neurocomputing, 456:
489,480, October 2021a. ISSN 09252312.
doi: 22.00666.ecccmm22220...999 URL arXiv: 2007.10573.
http://arxiv. www.:/rggg/aa/aaaeeooooo. /200 10573
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021b.
Yuankun Zhu, Yueqiang Cheng, Husheng Zhou, and Yantao Lu. Hermes Attack: Steal (DNN) Models with Lossless Inference Accuracy. 2021. ISBN 978-1-939133-24-33 URL https
zhu
A APPENDIX
A.1 DETAILS CONSTRUCTED MODELSET
We construct two modelsets (PACS-modelset and MEDU-modelstt) by enumerating combinations of attribute values. The architecture of each model modelsets follows the scheme: convolution layers, M fully-connected layers and a linear classifier. Each convolution layer contains a k k convolution, an optional batch normalization, an optional max-pooling and a non-linear activation function in sequence, where & is the kernel size. Each fully-connected layer consists of linear transformation, a non-linear activation and an optional dropout in sequence. We set the dropout ratio 0.1 in our experiments. When training models, optimizers selected from (SGD, ADAM, RMSprop} with a batch size 32, 64 or 128, respectively.
A.2 DETAILED IMPLEMENTATION MDGAN AND META- HEFAS-CLASSIFER
The MDGAN is composed of generator and multiple discriminators. The generator consists of two linear layers with ReLU activation. The dimension of the input layer of the generator is determined by the query number N and class category number C. In the experiment of recognizing handwritten digits, the input dimension is 1000 (N = 100, C' = 10). In the case PACS dataset, the input dimension is 700 (N 100, C 7), and the output dimension of the successive two layers is respectively 500 and 128. Each discriminator consists of three linear layer, with ReLU activation and final Sigmoid activation. The output dimension of layers are 512, 256, 1 respectively. There are meta-clase classifiers as total. Each meta-classifier is composed of two layer MLP with dimension of 128, 64, and the length of attribute values.
A.3 TRAINING STRATEGY ALGORITHM
The training strategy of DREAM is shown in Algorithm I
A.4 EXPERIMENTS DIFFERENT TRAINING AND TESTING ATTRIBUTES
We study the case that the white-box model and the black-box model to be inferred have completely different attributes. As we mentioned in Section 4.1, there are 5, 184 combinations of model attributes in total. We randomly sample 3000, 1000, 1000 training, validation and testing sets. None of the models has identical attributes. As shown in Table DREAM consistently outperforms other baselines the above setting.
12

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Training Strategy
Input: Batch size b, learning rate B, multi-domal model outputs O, trade-off scalar
Output: Generator G, meta-classifier h discriminators { Initialize: Initialize parameter 0g of generator G, parameter 03 of discriminators {Di J and parameter of domain-free meta-classifier with normal distribution
while classifier & not converges do Random sample samples Ož from outputs O³ in each domain
for L..., 772 do
Take samples in the ith domain True samples X = =O {x², ,..,,"}
for j j= 1,.., m and j + do
Take samples in the jth domain as False samples x = 0z = ( x9)
end
Update the discriminator D' by gradient descent:
0 = fg Vo {Ei= [logD(G(x*)] + Ejft Ed= log(1 D(GG( -
end
Construct Xau x faau- zall: ..., x J and Zall = G(Xall) = {z 2220
bm Pall
Set the corresponding labels as Yaul = fVat Yant Fall Calculate gradient of 0g and Og by: T yall log(p(=āu))
grad. Vo.
>im hm N=
grady Vo Ejtz Ei [log(1
- (
yau (log(p(=()))
Update the classifier and generator G together:
0. = 0. 3 grade and 8y 0g - a gradg
end
Table 4: Model attribute classification accuracy (%) P of PACS-modelset. Red and blue indicate the best and second best performance, respectively.
Attributes #conv
Method
Avg
#act
#drop
#pool
#ks
#fc
#opt
#bs
#bn
Random
25.00
50.00
50.00
50.00
33.33
33.33
33.33
33.33
50.00
39.81
SVM
34.20
51.70
48.50
56.10
35.70
36.50
37.60
40.50
64.60
45.04
KENNEN
37.36
53.12
57.79
59.66
38.94
35.93
37.92
41.71
63.91
47.37
SelfReg
26.08
52.35
53.89
52.70
35.11
33.84
37.46
36.42
50.99
42.09
MixStyle
35.98
54.31
57.35
57.43
37.14
35.51
39.31
42.07
57.84
46.33
P
MMD
38.67
57.16
61.49
58.73
40.65
39.14
38.69
41.06
71.48
49.67
DREAM
39.68
57.61
64.48
60.79
40.78
40.10
43.54
43.80
72.42
51.47
13

---

Under review as a conference paper at ICLR 2023
66
80
6a
Accuracy
securad
60
Accuracy
20
faah Attributes
Attributes
Attributes
10
10
10
0.3
0.1%
c
0.1
oor
0.01
0.01
Figure 7: Sensitivity analysis of parameter À on PACS-modelset, From left to right, the results in the split, C split and S split are shown, respectively.
60
60
Normelized Accuracy
Romallyes Accuracy
15
85
45
pрos
Pk
Nomalized
30
15
2tn
Arg
100
150
200
250
300
350
400
100
150
200
250
300
350
400
100
150
200
250
300
350
400
Query Num
Query Num
Query Num
Figure 8: Performance against query number PACS-modelset From left right, normalized accuracies in the split, C split and S split are shown, respectively.
A.5 SENSITIVITY ANALYSIS
We study the sensitivity of the trade-off parameter À in our final loss function on the PACS-modelset. As shown in Figure 7 the results for each model attribute do not show evident fluctuation when changing X, suggesting that our proposed method not sensitive to the choices of À in a wide range.
A.6 QUERY NUMBER AND SIZE OF TRAINING SET ANALYSIS
Query Number Analysis. Moreover, study the performance of DREAM against the number of queries on PACS-modelset. Following (Oh et al. 2018), we use the normalized accuracy that is linearly scaled according to random choice. As shown in Figure & with the increase of query numbers, the average performance does not improve but fluctuate, which means more queries do not necessarily provide more information for DREAM framework.
Size of Training Set Analysis. We further study the performance of our method against the size PACS-modelset. As shown in Figure observe that the performance slightly fluctuates from size of 1K to 5K, and does not consistently increase when the size increases. We suspect it can be attributed to the difficulty of our problem for domain-free attribute inference of black-box model, and the nature of OOD problem, i.e., the noise level increases as the size of training set grows. is worth studying further.
of training
60
60
60
Normllized Accuracy
Normalized Accuracy
45
Nermallieed Accuracy
85
ppnol
Ødrop
30
15
atn
Avg
1K
24
3R
4k
6k of Training set
5k
7k
0k
9k
1k
2k
3K
4k
5k
6k:
7k
9k
1k
2k
3k
4k
5k
6k
7k
Size
Size of Training set
Size
Traininng
Figure 9: Performance against size of training set moCe-sodelset. From left right, normalized accuracies in the split, C split and S split are shown, respectively.
14

---

Under review as a conference paper at ICLR 2023
A.7 STATISTICS OF MODELSET
We represent the statistics of each attribute value in PACS-modelset (Table Table 7 and MEDU- modelset (Table Table The "Ratio" line represents the proportion of models with the attribute value in the whole set of models. The next four lines represent maximal, median, mean and minimal accuracy of models for the attribute value, respectively.
Table 5: Distribution of attributes in domain P (Photo) of PACS-modelset and classification accuracy of our method on the Photo validation set.
drop
poto
CUITY
Yes
Yes
Ratio
24.7
50.9 49.1
49.7
50.6
49.4
33.5
33.2
33.2
34.0
mas
71.3
70.9
72.8
68.7
72.8 71.3
72.4 72.8
72.8
72.4
70.5
72.8
72.4
72.8
71.1
72.4
mcan
58.0
59.7
58.4
55.7
58.2
57.7
57.9
58.0
58.6
57.3
58.2
58.6
57.0
59.6
58.3
56.1
min
26.0
26.2
26.2
25.6
25.6
26.0
25.6
25.8
26.2
25.6
26.0
25.8
25.6
26.0
25.8
25.6
alg
bs
bn
SGD
ADAM RMSprop
32
64
128
Yes
No
Ratão
32.4
34.0
33.7
33.6
33.9
32.6
49.8
50.2
72.4
70.9
72.8
71.3
72.8
72.4
68.3 72.8
modian
56.5
60.6
61.0
61.2
60.0
57.9
56.3 63.6
meam
min
25.8
26.2
25.6
26.2
25.6
25.8
25.8 25.6
Table 6: Distribution of attributes in domain C (Cartoon) of PACS-modelset and classification accuracy of our method on the Cartoon validation set.
drop
pool
CUITY
ReLU
ELU PRoLU
Tanh
Yes
No
Yes
maa
73.0
71.0
71.7
69.9
71.7
73.0
71.6 73.0
73.0 71.7
73.0 71.9
71.7
73.0
71.9
70.7
median
63.3
62.3
62.8
62.0
61.6 63.5
63.0 61.7
63.2
63.0
61.4
mean
60.5
61.7
61.1
59.4
61.2
60.2
60.6 60.8
61.5 60.1
60.1
61.2
61.0
61.8
60.9
59.4
25.7
alg
bs
bn
SGD
ADAM RMSprop
32
64
128
Yes
No
Ratio
30.5
36.1
33.4
34.4
34.0
31.6
46.4
53.5
71.6
71.9
73.0
73.0
71.7
71.7
70,9
73.0
median
61.0
62.8
62.9
63.0
62.3
61.6
59.9
64.5
59.0
61.3
61.6
60.9
59.6
58.0
63.1
mio
25.1
27.2
28.8
25.8
25.8
25.1
25.1
28.4
Table 7: Distribution of attributes in domain S (Sketch) of PACS-modelset and classification accuracy of our method on the Sketch validation set.
drop
ReLU
BLU
PReLU
Tanh
Yes
No
Yes
5
Ratio
25.7
27.3
25.8
21.2
51.5
48.5
49.3
50.7
50.7 49.3
34.7
33.5
31.8
349 32.9
32.3
67.4
66.5
mediam
55.8
56.3
55.8
54.0
56.9
542
54.7
56.9
56.4
54,8
$3.5
56.1
57.3
56.0
55.9
55.1
mean
51.9
54.4
53.8
$2.0
$4.9
52.2
52.5
54.7
54.6
52.6
$2.0
53.9
55.1
56.0
53.7
52.8
25.1
25.2
25.1
25.1
25.1 25.1
25.1
25.1
25.1
25.1
25.1
alg
bs
bn
SGD
ADAM RMSprop
32
64
128
Yes
No
Ratio
32.0
35.8
32.2
34.3
33.6
32.1
46.0
max
63.5
67.4
66.5
67.4
65.7
65.8
64.8 67.4
median
54.0
56,9
57.1
54.9
54.7
55.1
51.5
55.6
53.4
55.2
52.8
52.7
53.6
$1.5
min
25.1
25.1
25.1
25.1
25.1
25.1
25.1
25.1
15

---

Under review as a conference paper at ICLR 2023
Table 8: Distribution of attributes in domain M (MNIST) of MEDU-modelset and classification accuracy of our method on the MNIST validation set.
act
drop
pool
ks
cOnY
ReLU
ELUU
PReI.
Tanh
Yes
No
Yes
5
2
3
Ratio
24.7
25.8
24.5
25.0
50.9
49.1
50.3 49.7
50.6
49.4
33.5
33.3 33.2
33.2
32.8
34.0
99.3
99.2
99.3
99.2
99.3
99.3
99.3 99.2
99.2 99.3
99.1
99.2
99.3
99.3
99.2
99.3
median
98.6
98.6
98.6
98.4
98.5
98.6
98.5 98.6
98.5 98.6
98.4
98.6
98.6
98.3
98.2
98.3
98.4
98.5
98.4
98.
98.3
98.5
98.3 98.4
98.3 98.5
98.4
98.6
98.6
98.5
98.5
min
91.6
94.3
36.9
73.5
36.9
63.8
63.8
36.9
63.8
36.9
63.8
91.6
36.9
92.8
63.8
36.9
bs
SGD
ADAM RMSprop
32
64
128
Yes No
Ratão
32.4
34.0
33.6
33.5
33.9
32.5
49.7 50.3
max
99.2
99.3
99.3
99.3
99.2
99.1
modian
98.4
98.6
98.5
98.6
98.5
98.5
98.4 98.7
mean
98.1
98.5
98.4
98.5
98.5
98.5
98.2 98.6
36.9
92.8
90.0
90.0
92.8
36.9
36,9
63.8
Table 9: Distribution attributes in domain E (EMNIST) of MEDU-modelset and classification accuracy of our method the EMNIST validation set.
drop
pool
CORV
ReLU
FLU
PRel.U
Tanh
No
Yes
No
Yes
3
5
2
3
2
4
Ratio
24.7
25.9
24.5
25.0
$1.0
49.0
50.3
49.7
50.5
49.4
33.6
33.2 33.2
33.2
32.8
34.0
99.6
99.6
99.6
99.6
99.5
median
99.1
99.1
99.1
98.9
99.1
99.0
99.0 99.1
99.0
99.1
98.9
99.1
99.1
99.1
99.1
99.0
98.8
min
35.7
87.8
31.9
31.9
31.9
33.4
79.2 31.9
31.9 31.9
94.5
40.6
31.9
94.7
41.3
31.9
RMSppop
Ratio
33.6
34.0
49.7
max
99.6
99.6
99.5
99.5
99.6
99.5
99.5 99.6
median
98.7
99.1
99.
99.1
99.0
98.9 99.1
mean
98.2
99.0
99.0
99.0
98.8
98.4
98.4 99.0
Table 10: Distribution of attributes in domain D (DIDA) of MEDU-modelset and classification accuracy of our method on the DIDA validation set.
drop
ReLU
BLU
PReLU
Tanh
Yes
No
Yes
5
Ratio
24.7
26.3
24.8
24.1
51.0
49.0
50.2
49.9
50.9 49.1
34.2
33.3
32,5
31.9
32.8
33,3
98.6
94.0
mediam
94.0
94.4
94.5
93.7
93.9
94.3
94.0 94.3
93.9 94.4
93.0
94.5
94.8
94.3
94.1
94,0
mean
92.7
93.1
93.2
92.0
92.8
92.7
92.9
92.6
92.3 93.2
91.6
93.2
93.6
93.2
92.7
92.3
25.0
25.2
25.0
25.1
32.7
25.0
25.0 25.0
26.0
25.1
26.5
25.1
alg SGD ADAM RMSprop
bs
bn
32
64
128
Yes
No
Ratio
31.7
35.4
32.9
33.9
33,9
32.1
47.6 $2.4
max
98.0
97.7
98.1
97.9
98.1
97.9
96.6
98.1
median
93.2
94,3
94.6
94.2
93.5
93.3
90.4
93.9
93.8
93.8
93.0
91.3
91.1
94.2
min
25.0
68.8
43.2
26.1
25.1
25.0
25.0
43.2
16

---

Under review as a conference paper at ICLR 2023
Table 11:
Distribution of attributes in domain U (USPS) of MEDU-modelset and classification accuracy of our method on the USPS validation set.
act
drop
pool
ks
COnY
ReLU
ELU
PReIU
Tanh
Yes
No
Yes
5
2
3
2
Ratio
25.1
26.2
25.0
23.7
51.0
49.0
49.1
50.9
50.9
49.1
33.5
33.4
33.1
33.3
32.7
34.0
max
98.9
98.9
98.8
98.7
98.8
98.9
98.9 98.8
98.8
98.9
98.5
98.7
98.9
98.9
98.8
98.9
median
97.2
97.1
97.2
96.9
97.1
97.1
96.8
97.3
96.9 97.3
96.5
97.3
97.5
97.1
97.1
97.1
96.5
96.3
96.6
94.9
96.4
95.8
95.3
96.8
95.8 96.4
94.8
96.5
96.2
96.3
95.8
min
30.4
30.5
29.6
26.1
26.1
26.7
26.7
26.1
26.7
26.1
28.5
26.7
26.1
36.1
26.7
26.1
bs
SGD
ADAM RMSprop
32
64
128
Yes No
Ratão
32.9
34.8
32.3
33.5
34,0
32.5
48.7 51.3
max
98.9
98.9
98.8
98.9
98.6
modian
96.9
97.3
97.2
97.3
97.1
96.8
98.6 98.9
mecm
96.1
96.5
95.7
96.8
96.3
95.2
96.7 95.5
26.1
29.7
26.7
31.8
26.7
26.1
26.1 26.7
A.8 EXPERIMENT OF MORE REALISTIC SCENARIO
We subset of these K classes for training. when training the white box model, leave the "dog" and "elephant" for each domain. Then, we train domain-free meta classifiers using model outputs without the "dog" and "elephant" classes and then test domain-free meta classifiers using model outputs that hold all classes. As shown in the 12 our method can still perform well compared with baselines.
Table 12: Model attribute classification accuracy (%) on moddeee. Red and blue indicate the best and second best performance, respectively, DREAM represents that its domain-free meta classifier trained on model outputs without the "dog" and "elephant" classes and tested on model outputs that hold all classes.
Attributes
Method
Avg
#act
#drop
#pool
#ks
#conv
#Cc
#opt
#bs
#bn
Random
25.00
50.00
50.00
50.00
33.33
33.33
33.33
33.33
50.00
39.81
SVM
23.80
47.60
47.40
45.80
33.80
34.50
31.80
34.30
53.10
39.12
KENNEN *
34.64
50.10
53.07
52.01
34.61
37.11
35.78
37.04
55.27
43.29
SelfReg
27.07
54.32
51.39
53.07
36.99
36.82
35.47
34.17
61.80
43.46
MixStyle
37.78
51.71
54.16
53.60
34.53
36.16
36.36
36.02
59.42
44.42
S
MMD
31.96
52.94
56.84
52.78
38.18
38.20
36.20
35.92
57.56
44.51
DREAM
42.24
55.68
61.82
58.34
39.55
38.39
38.51
41.39
74.39
50.03
DREAM
39.71
57.74
64.73
60.79
40.79
40.14
43.54
43.80
72.51
51.53
A.9 EXPERIMENT OF APPLICATION OF REVERSE ENGINEERING
Let us consider the setting of model extraction. The structure of the target model is unknown. We use an arbitrary random network structure to extract the target model with the method DFME [1], and use the structure inferred by our method to extract the target model. As shown in the above table, the experimental result shows that using the structure inferred by method obtains better extraction performance, indicating that our findings are significant.
[1] Kariyappa, S., Prakash, A., & Qureshi, M. K. (2021). Maze: Data-free model stealing attack using zeroth-order gradient estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13814-13823).
17

---

Under review as a conference paper at ICLR 2023
Table 13: Accuracy and normalized accuracy of data-free model extraction methods. The structure of student model has three choices, same to victim, randomly generate ten structures and compute average, predicted by DREAM. Results for "DREAM predict" reflect our support for model extraction task in black box setting.
Structure of student model
Dataset
Victim accuracy(black box model)
#same to victim #Random 10)
#DREAM predict
MNIST
86.43%
68.46%((u.79x)
45.88% (0.53x)
62.81 (0.73x)
18


Reviews:
[{"review_id": "nYiF0O3TEI", "reviewer": "Reviewer_iGeV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets a new setting of domain-free reverse engineering the attributes of black-box models (DREAM) and casts it as an out of distribution (OOD) generalization problem. In particular, a multi-discriminator generative adversarial network (MDGAN) is proposed to learn domain invariant features on the training data that consists of multiple domains. Then a domain-free reverse model is learnt on domain invariant features, which can be generalized to infer the attributes of black-box model with a new domain. Extensive experimental studies are conducted to compare the proposed method with various baselines.", "summary_of_the_review": "The DREAM problem is somehow interesting. The empirical support is sufficient. However, I have concerns about the novelty and correctness of the techniques. In addition, I think the conducted case is somehow restricted and the technical motivation of this work is not significantly strong.", "strength_and_weaknesses": "The DREAM problem is somehow interesting. The empirical support is sufficient. However, I have the following concerns:\n\nWeakness:\n1) This work has limited technical contributions. The proposed method is simply an extension of adversarial domain adaptation (ADA) [1], which also learns representations invariant to the domains via adversarial training for the downstream task. The authors may argue the difference between a single discriminator and multiple discriminators, which is actually a marginal modification. Even, the multiple discriminators require larger computation. I think it is more efficient to devise a single discriminator with multi-classification on domain labels.\n\n2) The technical part exists severe mistakes. The domain-free reverse classifier should target the classification on attributes of the model instead of the data labels. Eq. (4) and Eq. (5) are incorrectly defined.\n\n3) The OOD generalization only considers the domain shift, which is somehow restricted. As there is no access to the target model’s dataset, the training data may deviate significantly from the target dataset, such as having completely different labels than the desired dataset. Exploring this would be more interesting.\n\n4) There is no introduction to the OOD baselines. I think the authors should also analyze the challenges or issues when directly applying the OOD baselines to the target problem from technical principles in addition to empirical support.\n\n5) Some experimental setups/analyses/results are confusing.\nI. What do you mean “We also sample 5000, 1000, 1000 from white-box models as the training set, validation set, and testing set”.\nII. “In the scenery of easier cases, e.g., #act, #ks, #fc in M of MEDU (as shown in Table 3), our proposed DREAM is more likely to overfit due to more parameters, degrading the performance of our method.” How to understand it? What is “more parameters” meaning?\nIII. T-SNE of DREAM looks a bit strange and seems unstructured, i.e., “a long line”.\n\nReferences\n[1] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., ... & Lempitsky, V. (2016). Domain-adversarial training of neural networks. The journal of machine learning research, 17(1), 2096-2030.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mNEEHOYHXG", "reviewer": "Reviewer_W21i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to reverse engineer the attributes of black-box neural networks without knowing the training set of the target model. Specifically, this paper transforms the reverse engineering problem into an out of distribution generalization problem and constructs a DREAM framework to predict the attributes of a black-box model. With MDGAN, the author demonstrates it is able to learn domain invariant features in the scenario of attribute inference of black-box mode.", "summary_of_the_review": "Although there is room for further improvement in the practical application of the proposed framework, the new setting and novel framework proposed in this paper can provide deep insights into the problem of neural network reverse engineering. The model dataset constructed in this paper and large-scale experiments can also serve as important references for subsequent work. I am generally optimistic about this submission.", "strength_and_weaknesses": "Strength\n\nThe setting that training dataset is not available is practical.\n\nConstruction of model dataset by enumerating all possible attribute values and all possible combinations.\n\nThe experiments are extensive using a variety of models and data， including comparison with baseline methods.  Analysis on diverse modelsets is insightful.\n\nThe proposed OOD generalization learning idea can benefit future research.\n\nDomain-free reverse engineering towards the attributes of black-box model is an exciting and novel research problem.\n\n\nWeaknesses\n\nThe contribution of this paper can be further improved if the potential practical use of the proposed DREAM can be elaborated.\nIt is not clear to me about the novelty of the MDGAN.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RVEOsWfm6CZ", "reviewer": "Reviewer_p3er", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The paper tackles the problem of reverse-engineering attributes (e.g., #conv. layers? Optimizer?) of a black-box model.\n- The main motivation is that existing work (Oh et al., '18) assumes the attacker has the same labeled dataset as that used to train the victim black-box model. The paper proposes an approach that overcomes this limitation by using OOD techniques. Specifically, the proposed approach learns a domain-invariant embedding which is subsequently input to the attribute classifier.\n- Results are validated on two datasets (PACS and MEDU) and compared with Oh et al. '18 along with other relevant OOD baseline approaches.", "summary_of_the_review": "I find that the paper advances the state of the black-box attribute reverse-engineering problem, by using a reasonable technical insight and further demonstrating reasonable performance gains over baselines. However, I am not entirely convinced how significant the findings are, given reasonably strong assumptions (e.g., fully labeled data from multiple domains) and the relevance (how useful is ~50% accuracy in determining simple attributes, when random guessing is ~40% accurate). ", "strength_and_weaknesses": "### Strengths\n\n**1. Evaluation and Results**\n- I appreciate the manner in which the approach is evaluated -- specifically, the paper considers numerous OOD baselines along with Kennen (Oh et al., '18). Many of the proposed baselines themselves exhibit marginal improvements.\n\n**2. Insight to Approach**\n- The approach uses an intuitive strategy: to treat generalizing to new test-time inferences as a domain-shift problem in the meta-classifiers input space. The authors also present an experiment (Figure 5) to validate that the domain-shift is addressed.\n\n### Concerns\n\n**1. Proposed threat model makes marginally weaker assumptions**\n- The paper does take a step in the right direction, by showing that the reverse-engineering attribute attack is possible in spite of the attacker not possessing the victim's dataset. However, if I understand right, the attacker  needs to nonetheless possess a fully labeled dataset, with multi-domain images per label. Moreover, this dataset needs to be large enough to train thousands of different models. As a result, it appears that the paper somewhat strong assumptions nonetheless.\n\n**2. Consequences of reverse engineering attributes**\n- Another concern I have is the significance of findings. Although reverse-engineering attributes highlights that confidential IP can be uncovered, it is much weaker in light of e.g., recent model extraction attacks. This is further exacerbated with mediocre improvements over random guessing -- while it is definitely an improvement over previous works, it appears far from perfect (e.g., ~48-57% accuracy, compared to ~40% random accuracy). Could the authors highlight the significance of the findings?\n- One potential solution I see is exploiting this information as a prior for complementary attacks. For instance, crafting adversarial examples as show in Oh et al., '18.\n\n**3. Writing**\n- I found a few paragraphs and figures difficult to follow and would appreciate if the authors revised them.\n- (a) Notation (esp. in Sec. 3.3): it took me many reads to understand what exactly are the outputs $O^i_j$ e.g., \"$f_i$ consists of $K$ models of $i$-th domain - what are the $K$ models? how many are domains are there? \n- (b) Sec 3.5: Is there a single reverse classifier $\\Phi$? (I thought there is a separate classifier for each attribute?) What is the architecture of $\\Phi$? Is the input $z^i$ to the classifier a $R^{K \\times d'}$ tensor (with $K$ = 5000?)? \n- (c) Fig. 3: In the left yellow block, the multi-domain query (3 images) is input to each of the domain models? Rather, I think each of the domain classifiers has inputs only from a single domain?\n- (d) (Nitpicks) Numerous typos: \"When feed them\", \"in the scenery of complicated data\", ... Please make another pass over the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cr_WUPumKzi", "reviewer": "Reviewer_sNFK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates how to predict attributes of black-box models (e.g., provided through a prediction API). Unlike previous work, the authors investigate the more realistic case where the training set is unknown. Adversarial domain generalization tools are used to learn a domain-free reverse engineering classifier for black-box model attributes. In experiments, this outperforms the previous method that does not consider domain shift.", "summary_of_the_review": "Due to the limited novelty of the paper, I cannot recommend acceptance at this time. However, I may have misunderstood some aspect of the paper and would be willing to reconsider based on the author response.\n\n-----------\nUpdate after rebuttal:\n\nThe authors addressed my technical concerns, but I'm still not entirely convinced of the novelty of the paper. Still, this is a solid improvement/extension over (Oh et al., 2018), so I'm now leaning towards acceptance.", "strength_and_weaknesses": "Strengths:\n- A useful extension of (Oh et al., 2018) to a more realistic setting\n- The writing is mostly clear and the paper is structured well\n- Results show that the proposed DREAM method outperforms the previous method that does not consider the possibility of domain shift in the training set of the black-box model\n\nWeaknesses:\n- Limited novelty; this is an X+Y paper where X = the reverse engineering approach from (Oh et al., 2018) and Y = adversarial domain generalization\n- The paper claims to be concerned about realism of the setting, but the experiments on PACS run somewhat counter to this. Surely one would know whether a black-box model being stolen is trained on cartoons vs photos, and thus be able to collect a dataset for training the white-box models that consists of the proper domain. A more realistic distribution shift would be leaving out certain modes of the data, or certain classes.\n- Serious missing citation: \"Domain-Adversarial Training of Neural Networks\". This is a well-known domain generalization paper using a similar idea to the one proposed in this paper, so it should be cited and compared to.\n- Are the SelfReg, MixStyle, and MMD baselines applied to KENNEN? This isn't clear.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject