Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a Dual Student (DS) to mimic the behavior of a black-box model without any prior about input data. The main advantage of DS over existing work lies in that it allows a naturally differentiable process for the training of generator. 
strength_and_weaknesses: Pros:
1. The introduced additive student model provides an alternative way to update generator which looks reasonable and convincing.
2. The derived gradients are experimentally demonstrated closer to the true gradient (computed by target model) than DFME which only uses a single student.
3. The experiments showed that DS can approximate the target model more accurately than the baselines across various tasks.

Cons:
1. Is it possible that two student behaves like two experts in MoE. Let’s say two students work in a complementary way. Will this be risky for simply using the average output during inference? Any experimental observation about this concern?
2. The authors mentioned that Hong el al. maximized the confidence of the student output. According to my experience, this insight might be helpful to learn a data distribution that is close to the real training data. Has this method been included as a baseline?
3. Following 2, I felt that data-free model stealing has some overlap with the topic of black-box inverse attack which aims to steal the training samples. The authors should discuss their relations. Particularly, according to the presented gradient error (fig 2), I suspected that the generated data by DS or DFME does NOT looks similar to the true training samples.
4. I agreed that the proposed formulation may benefit the training of generator. However, the claim of exploring diverse training data has not been well supported. Any further evidence about it?
5. It would be perfect if some theoretical analyses are provided to support the distance of Eq. 7 is minimized by DS.
6. Please check the equation number, some of which are not properly referred.
clarity,_quality,_novelty_and_reproducibility: The idea is novel. The writing needs improving as many sentences are too long to read, and some grammar mistakes should be corrected as well.
summary_of_the_review: This paper is overall good. The writing needs improving as many sentences are too long to read, and some grammar mistakes should be corrected as well. I expect the authors’ response to my above comments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies how to perform model stealing in the data-free setting.
The authors first identify two weaknesses of existing data-free model stealing methods: (1) the generated query data are not informative enough, and (2) the estimation of black-box gradients is not stable for optimization.

To address these two issues, the authors then design a novel dual-student model stealing method.
The dual-student method, which involves two student networks and a query data generator, aims to solve a minimax problem. The two student networks aim to imitate the behavior of the victim model on generated query data distribution, while the generator aims to generate query data that maximize the prediction differences between student networks and the victim.
Through a simple substitution, the solution of the maximization problem can be approximated as maximizing the disagreement between the two student networks, which therefore does not require black-box optimization anymore.
strength_and_weaknesses: The proposed dual-student method is interesting and empirical results show that it surpasses existing data-free approaches in some situations. However, I think the current version of the paper can be improved in many aspects, which are listed below:

1. The writing of the paper is not well, making it a little confusing to follow the ideas behind the paper. Specifically, in Section 1, the authors put most of the details of dual-student in a single paragraph, which makes it difficult for reading. Besides, Section 2 uses too much content in reviewing previous works (more than 1 page), which however is not so necessary for introducing the proposed dual-student methods.

2. The insight of the proposed dual-student method is not clear enough.
    - Why replacing the objective problem in Eq.(1) with Eq.(2) is appropriate? I suggest the author give at least an intuitive explanation about it.
    - An ablation study on the relationship between the step numbers for updating the generator and student networks would help understand why the dual-student method performs well in practice.

3. Some experiment details are missing.
    - What is the model architecture of the query data generator?
    - Please explain how to combine the two methods, DFMS-HL/SL and DS in detail.

4. The authors claim that they conduct experiments on the CIFAR-100 dataset. However, I could not find corresponding experiment results in tables or figures from the paper.

5. Some suggestions about improving the empirical studies.
    - Study the relationship between query budget and stealing performance of the dual-student method.
    - Involve more black-box attacks for evaluation, e.g., membership inference attacks, and model-inversion attacks.

6. Typos: In Eq.(3): $G(x)$ -> $G(z)$.
clarity,_quality,_novelty_and_reproducibility: The proposed dual-student method is interesting, but the authors fail in conducting a comprehensive empirical study.
summary_of_the_review: In general, the proposed method is novel and shows some performance advantages. However, the authors did not fully justify its effectiveness. Besides, the writing of this paper needs to be improved. Therefore, I tend to reject this paper for now. I suggest the authors to improve the writing, add more experiments, and submit the paper to another top conference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper has addressed the problem of data free model stealing, where a dual student based framework is proposed for better estimating gradients of the target model without access to its parameters, and generating a diverse set of images that thoroughly explores the input space. While the proposed framework looks interesting, its design is not well justified and not well ablated.
strength_and_weaknesses: **Strengths**

(1) The paper addresses an interesting problem of data-free learning or data-free model stealing, where they proposed a dual-student based framework.

(2) The paper has reported encouraging results on the available benchmarks for both standard classification accuracy and transfer-based attack effectiveness.

**Weaknesses**

(1) It is not clear why a dual student model is needed for this. How are they creating more effective criteria? In my understanding it can also be done with the target model and only one student. In that case, the only student will be trained to match the target model, while the generator can be tasked to generate samples on which the student and the target model disagree for creating hard samples. More details with writing is needed together with appropriate ablation study.

(2) There are some clarifications on the reasons at the very last paragraph of section 3.1, however those reasoning aren't well grounded. So please cite the papers that utilize the forward differences method to estimate the gradient through the target model. Additionally, ablative studies should be performed or indicated to justify the design choice.

(3) Results on CIFAR 100 dataset are missing in table 1. As the model is motivated based on its effectiveness of generating more diverse samples to be effective for datasets with more number of classes, results on CIFAR 100 is important. In table 1 of the supplementary material, both the rows show results on CIFAR 10, is it correct? If so, why are the numbers in the DFMS-HL/SL rows different?
clarity,_quality,_novelty_and_reproducibility: The paper is reasonably clear. It is well written in good English and the reported results are encouraging. The authors haven't provided the code, however I expect it will be available at some point facilitating the reproducibility of the method. While the model is interesting, certain design choices are not clear or well ablated and some important experiments (CIFAR 100) are missing.
summary_of_the_review: The paper is interesting with a new architecture, but the design choices are not well grounded. Some important experiments are missing which makes it difficult to judge the effectiveness of the proposals.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper adresses data-free model stealing, an ambitious framework where one attempts to build a model that imitates a target one without accessing the target parameter or any training data point (true or substitute ones). In this field, SOTA approaches use a student network whose outputs are meant to minimize a distance between its outputs and the outputs of the target (which is the only information unveiled by the target model). The student competes with a an adversary as part of a min-max game in order to efficiently explore the input space. 
 
Compared to previous approaches, the authors use two students and exploit disagreement between the two to better explore the input space in regions where harder inputs live. Disagreement is exploited by an adversary input generator which will try to maximize conflict. The generated inputs are the one on which student and teacher prediction should align. 
strength_and_weaknesses: Pros : 
- the paper is well written and pleasant to read
- the proposed contribution requires fewer queries to the to-be-stolen teacher model and is also applicable in situations where the number of classes is not known in advance. 
- the proposed approach is fully differentiable which is a nice increment over SOTA.

Cons : 
- the paper does not mention computation time/complexity overhead incurred by using an additional student
clarity,_quality,_novelty_and_reproducibility: The paper is clear. The contributions are well motivated and fairly new. 
The authors provide a reproducibility guidelines in the paper.
summary_of_the_review: Major remarks: 

The only missing aspect in the paper is an analysis of increased time/memory requirements of the method  compared to prior arts.

Let alone that, the paper is interesting and I do not have compelling remarks to address but rather a few questions that came to my mind :

Is there any benefits in training more than 2 students ? Perhaps more precise finer-grained disagreement regions could be obtained in this way.

Do the author have an opinion on possible counter-measures that a defender could implement to misguide the students ?

Minor :

In some parts of the paper, the authors use the word « image » as if it was obvious that inputs are images.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission tackles the problem of data-free model stealing using an approach called Dual Students. It consists of a generator network and two student networks, in addition to the black-box target network. The generator network is tasked with producing model inputs for which the student network predictions disagree, and the student networks are tasked with matching the target network's output on the generator-produced examples. In contrast to previous work where the generator is tasked with maximizing the disagreement between a student network and the target network (thereby requiring to compute the target network's Jacobian), Dual Students is fully differentiable with respect to the generator's loss.

The proposed approach is evaluated in terms of student classification performance on MNIST, FashionMNIST, GTSRB, SVHN, and CIFAR10, and is compared against DFME and DFMS-HL. It is shown to outperform competing approaches in that setting. The submission also presents black-box transfer-based adversarial attack results, showing that data-free methods like the proposed approach are more successful than using a proxy model trained on the real dataset and also outperform a white-box FGSM attack.

Reviewers note that the submission proposes a reasonable and convincing idea (vQa1) which is novel (nQa1, xQYc, aGyH) and addresses an interesting problem (yxtQ). All reviewers also agree in one way or another that the empirical results presented look promising. Some reviewers had initial concerns regarding writing quality (vQa1, xQYc), the relationship to black-box inverse attacks (vQa1), the intuition behind proposed approach (xQYc), missing experimental details (xQYc), and missing CIFAR100 results (xQYc) which were addressed to their satisfaction by the authors.

Reviewers are split on acceptance, with Reviewers vQa1 and aGyH for it and reviewers xQYc against it. Reviewer xQYc remains concerned that the query efficiency results presented are insufficient, because they are restricted to one combination of baseline method, experiment setting, and dataset. Somewhat less importantly, they also remain concerned that the submission is missing evaluations under more attacks.

Ultimately, the majority opinion is that the submission's merits outweigh its flaws.
justification_for_why_not_higher_score: There remains a concern that the submission is lacking sufficient query efficiency analyses.
justification_for_why_not_lower_score: The submission stands on its own given its merits (new, reasonable, and convincing approach to data-free model stealing).
Decision: Accept: poster