Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates how to predict attributes of black-box models (e.g., provided through a prediction API). Unlike previous work, the authors investigate the more realistic case where the training set is unknown. Adversarial domain generalization tools are used to learn a domain-free reverse engineering classifier for black-box model attributes. In experiments, this outperforms the previous method that does not consider domain shift.
strength_and_weaknesses: Strengths:
- A useful extension of (Oh et al., 2018) to a more realistic setting
- The writing is mostly clear and the paper is structured well
- Results show that the proposed DREAM method outperforms the previous method that does not consider the possibility of domain shift in the training set of the black-box model

Weaknesses:
- Limited novelty; this is an X+Y paper where X = the reverse engineering approach from (Oh et al., 2018) and Y = adversarial domain generalization
- The paper claims to be concerned about realism of the setting, but the experiments on PACS run somewhat counter to this. Surely one would know whether a black-box model being stolen is trained on cartoons vs photos, and thus be able to collect a dataset for training the white-box models that consists of the proper domain. A more realistic distribution shift would be leaving out certain modes of the data, or certain classes.
- Serious missing citation: "Domain-Adversarial Training of Neural Networks". This is a well-known domain generalization paper using a similar idea to the one proposed in this paper, so it should be cited and compared to.
- Are the SelfReg, MixStyle, and MMD baselines applied to KENNEN? This isn't clear.

clarity,_quality,_novelty_and_reproducibility: The novelty is limited (see above). The quality of the writing is good, and the quality/reproducibility of the experiments is good.
summary_of_the_review: Due to the limited novelty of the paper, I cannot recommend acceptance at this time. However, I may have misunderstood some aspect of the paper and would be willing to reconsider based on the author response.

-----------
Update after rebuttal:

The authors addressed my technical concerns, but I'm still not entirely convinced of the novelty of the paper. Still, this is a solid improvement/extension over (Oh et al., 2018), so I'm now leaning towards acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper targets a new setting of domain-free reverse engineering the attributes of black-box models (DREAM) and casts it as an out of distribution (OOD) generalization problem. In particular, a multi-discriminator generative adversarial network (MDGAN) is proposed to learn domain invariant features on the training data that consists of multiple domains. Then a domain-free reverse model is learnt on domain invariant features, which can be generalized to infer the attributes of black-box model with a new domain. Extensive experimental studies are conducted to compare the proposed method with various baselines.
strength_and_weaknesses: The DREAM problem is somehow interesting. The empirical support is sufficient. However, I have the following concerns:

Weakness:
1) This work has limited technical contributions. The proposed method is simply an extension of adversarial domain adaptation (ADA) [1], which also learns representations invariant to the domains via adversarial training for the downstream task. The authors may argue the difference between a single discriminator and multiple discriminators, which is actually a marginal modification. Even, the multiple discriminators require larger computation. I think it is more efficient to devise a single discriminator with multi-classification on domain labels.

2) The technical part exists severe mistakes. The domain-free reverse classifier should target the classification on attributes of the model instead of the data labels. Eq. (4) and Eq. (5) are incorrectly defined.

3) The OOD generalization only considers the domain shift, which is somehow restricted. As there is no access to the target model’s dataset, the training data may deviate significantly from the target dataset, such as having completely different labels than the desired dataset. Exploring this would be more interesting.

4) There is no introduction to the OOD baselines. I think the authors should also analyze the challenges or issues when directly applying the OOD baselines to the target problem from technical principles in addition to empirical support.

5) Some experimental setups/analyses/results are confusing.
I. What do you mean “We also sample 5000, 1000, 1000 from white-box models as the training set, validation set, and testing set”.
II. “In the scenery of easier cases, e.g., #act, #ks, #fc in M of MEDU (as shown in Table 3), our proposed DREAM is more likely to overfit due to more parameters, degrading the performance of our method.” How to understand it? What is “more parameters” meaning?
III. T-SNE of DREAM looks a bit strange and seems unstructured, i.e., “a long line”.

References
[1] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., ... & Lempitsky, V. (2016). Domain-adversarial training of neural networks. The journal of machine learning research, 17(1), 2096-2030.

clarity,_quality,_novelty_and_reproducibility: 1. The proposed method has a limited novelty.
2. The paper exists severe mistakes in the method part.
3. The technical motivation is not clearly clarified.
4. Some parts of the experiments are clarified not clearly.
summary_of_the_review: The DREAM problem is somehow interesting. The empirical support is sufficient. However, I have concerns about the novelty and correctness of the techniques. In addition, I think the conducted case is somehow restricted and the technical motivation of this work is not significantly strong.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - The paper tackles the problem of reverse-engineering attributes (e.g., #conv. layers? Optimizer?) of a black-box model.
- The main motivation is that existing work (Oh et al., '18) assumes the attacker has the same labeled dataset as that used to train the victim black-box model. The paper proposes an approach that overcomes this limitation by using OOD techniques. Specifically, the proposed approach learns a domain-invariant embedding which is subsequently input to the attribute classifier.
- Results are validated on two datasets (PACS and MEDU) and compared with Oh et al. '18 along with other relevant OOD baseline approaches.
strength_and_weaknesses: ### Strengths

**1. Evaluation and Results**
- I appreciate the manner in which the approach is evaluated -- specifically, the paper considers numerous OOD baselines along with Kennen (Oh et al., '18). Many of the proposed baselines themselves exhibit marginal improvements.

**2. Insight to Approach**
- The approach uses an intuitive strategy: to treat generalizing to new test-time inferences as a domain-shift problem in the meta-classifiers input space. The authors also present an experiment (Figure 5) to validate that the domain-shift is addressed.

### Concerns

**1. Proposed threat model makes marginally weaker assumptions**
- The paper does take a step in the right direction, by showing that the reverse-engineering attribute attack is possible in spite of the attacker not possessing the victim's dataset. However, if I understand right, the attacker  needs to nonetheless possess a fully labeled dataset, with multi-domain images per label. Moreover, this dataset needs to be large enough to train thousands of different models. As a result, it appears that the paper somewhat strong assumptions nonetheless.

**2. Consequences of reverse engineering attributes**
- Another concern I have is the significance of findings. Although reverse-engineering attributes highlights that confidential IP can be uncovered, it is much weaker in light of e.g., recent model extraction attacks. This is further exacerbated with mediocre improvements over random guessing -- while it is definitely an improvement over previous works, it appears far from perfect (e.g., ~48-57% accuracy, compared to ~40% random accuracy). Could the authors highlight the significance of the findings?
- One potential solution I see is exploiting this information as a prior for complementary attacks. For instance, crafting adversarial examples as show in Oh et al., '18.

**3. Writing**
- I found a few paragraphs and figures difficult to follow and would appreciate if the authors revised them.
- (a) Notation (esp. in Sec. 3.3): it took me many reads to understand what exactly are the outputs $O^i_j$ e.g., "$f_i$ consists of $K$ models of $i$-th domain - what are the $K$ models? how many are domains are there? 
- (b) Sec 3.5: Is there a single reverse classifier $\Phi$? (I thought there is a separate classifier for each attribute?) What is the architecture of $\Phi$? Is the input $z^i$ to the classifier a $R^{K \times d'}$ tensor (with $K$ = 5000?)? 
- (c) Fig. 3: In the left yellow block, the multi-domain query (3 images) is input to each of the domain models? Rather, I think each of the domain classifiers has inputs only from a single domain?
- (d) (Nitpicks) Numerous typos: "When feed them", "in the scenery of complicated data", ... Please make another pass over the paper.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: Average. I had trouble with some parsing the exact input/outputs to different blocks in the model (see Concern 3).

**Quality**: Slightly below average, because I am not entirely convinced with the significance of findings (see concerns 1, 2).

**Novelty**: Good. The technical insight (domain generalization) used by the authors is well-motivated, and perhaps will motivate future attack studies (where lack of data can be circumvented by treating it as a domain generalization problem)

**Reproducibility**: Average. Although the code is not provided, I am reasonably confident in reproducing the results given the description in Appendix A1 and A2. One detail that is not clear to be is the architecture of the "reverse model".
summary_of_the_review: I find that the paper advances the state of the black-box attribute reverse-engineering problem, by using a reasonable technical insight and further demonstrating reasonable performance gains over baselines. However, I am not entirely convinced how significant the findings are, given reasonably strong assumptions (e.g., fully labeled data from multiple domains) and the relevance (how useful is ~50% accuracy in determining simple attributes, when random guessing is ~40% accurate).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to reverse engineer the attributes of black-box neural networks without knowing the training set of the target model. Specifically, this paper transforms the reverse engineering problem into an out of distribution generalization problem and constructs a DREAM framework to predict the attributes of a black-box model. With MDGAN, the author demonstrates it is able to learn domain invariant features in the scenario of attribute inference of black-box mode.
strength_and_weaknesses: Strength

The setting that training dataset is not available is practical.

Construction of model dataset by enumerating all possible attribute values and all possible combinations.

The experiments are extensive using a variety of models and data， including comparison with baseline methods.  Analysis on diverse modelsets is insightful.

The proposed OOD generalization learning idea can benefit future research.

Domain-free reverse engineering towards the attributes of black-box model is an exciting and novel research problem.


Weaknesses

The contribution of this paper can be further improved if the potential practical use of the proposed DREAM can be elaborated.
It is not clear to me about the novelty of the MDGAN.

clarity,_quality,_novelty_and_reproducibility: There is no problem with the quality of this article. I have no negative comments on the quality of the paper and the clarity of the method description. In general, the proposed DREAM framework is not difficult to reproduce.
summary_of_the_review: Although there is room for further improvement in the practical application of the proposed framework, the new setting and novel framework proposed in this paper can provide deep insights into the problem of neural network reverse engineering. The model dataset constructed in this paper and large-scale experiments can also serve as important references for subsequent work. I am generally optimistic about this submission.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors propose a method for domain-free reverse engineering of neural network attributes (such as the number of convolutional layers). This work extends that of Oh et al., 2019, which focused on identifying attributes of a model when the training set is known. The key idea is to extend the work of Oh with adversarial domain generalization. While the reviewers noted that the setting is practical, they also noted limited novelty relative to previous work.
justification_for_why_not_higher_score: While three reviewers voted for acceptance, they also agreed novelty and technical contributions remained relatively limited. While the paper indeed provides some insight, the limitations suggest that the paper could be stronger.
Decision: Reject