Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a novel method for performing model extraction attacks capable of not only retaining a good performance but also capturing the robustness of the target model when using adversarial training. The proposed attack, which can be applied under a very practical (and restrictive) threat model, creates “uncertain examples” for querying the victim’s model. These examples are designed so that they have quite uniform confidence scores across the different classes in the target classifier. The experimental results show the benefits of the proposed approach in keeping a good trade-off between accuracy and robustness when compared with existing state-of-the-art attacks. The experiments also provide a comprehensive analysis of the behavior of the proposed method with different configurations of datasets, models and types of adversarial training and attacks, which help to convince the readers about the benefits of the method. 
strength_and_weaknesses: Strengths: 
+ The paper is well motivated, organized and written. It reads very well. The description of the method and the rationale for crafting the “uncertain examples” are very clear and intuitive.
+ The way the attack is intended to retain the model’s robustness is novel. Yet some of the previous works are capable of retaining part of the victim’s model robustness, the proposed method is capable of maintaining a good trade-off between accuracy and robustness and it is capable of working relatively well with more restrictive threat models. 
+ The experimental evaluation is very comprehensive and helps to convince the reader about the benefits of the proposed approach and its limitations. In this sense, the authors present an honest work recognizing the need for investigating more advanced model extraction methods capable of reducing the gap between accuracy on clean and adversarial examples. Nevertheless, I think the paper provides interesting insights and can foster further research in this area. 

Weaknesses: 
+ Although the experimental evaluation is very comprehensive, I think that the results in Table 1 should be discussed further, highlighting more the good results obtained with BEST compared to the other state-of-the-art methods, especially taking into consideration that some of them rely on more restrictive assumptions than BEST.
+ I think that the discussion about the ability of BEST to bypass defenses should be included in the main paper rather than in the appendices (by, for instance, moving some of the results in 5.3 to the appendix). I think this is an important aspect of the attack that, perhaps, deserves to be better emphasized in the paper. 

clarity,_quality,_novelty_and_reproducibility: As mentioned before the paper is very well written and organized. The background and the related work are covered quite well. The description of the proposed method is well motivated and very intuitive. In the experiments, the experimental settings and the results are well detailed, which helps with the reproducibility of the results. 
In terms of novelty, the paper aims at analyzing the problem of model extraction attacks by taking into account robustness. Some of the previous works have analyzed this aspect, but not with the angle given in this paper.  

summary_of_the_review: I find the paper quite interesting and opens the door to the investigation of model extraction attacks capable of retaining both clean and adversarial accuracy. Although the experiments show that there is still a gap for retaining the adversarial accuracy, this paper proposes the first attack that is really designed to extract the robustness of the victim’s model while keeping the clean accuracy. It is also interesting to observe that the attack is also capable of bypassing some existing defenses, although the analysis with respect to this is shallower in the paper. Nevertheless, the comprehensive experimental evaluation is quite convincing to show the benefits (and limitations) of the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of extracting robust models by black-box queries to the model. The authors find that questing uncertain samples with respect to the current local model are helpful for extracting the decision boundary of the remote (black-box) victim model. Empirical results show that the proposed method can extract models with competitive clean and robust accuracy while existing baselines failed to preserve the robustness of the remote model after extraction.  
strength_and_weaknesses: Strength:
1. the problem stealing robust models is interesting and is of practical interest in future.
2. the proposed method outperforms existing baseline attacks significantly.
3. the evaluations are quite comprehensive. 

weakness:
1. the robustness of the extracted model is still very low and is not useful in practice. Also, the performance gap between the proposed approach and best performing baseline is not that significant. 
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and is easy to follow. The proposed idea of leveraging uncertain examples to effectively extract models is novel. 
summary_of_the_review: Overall, I like the idea of the leveraging uncertain samples for efficient model extraction. The empirical results also support the main claims in the paper well. The only concern here is, although the proposed method works better than the baselines, the robustness of extracted models are not high, and is of limited practicality. Therefore, a weak accept is recommended.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers a model stealing problem through remote access such as Rest API, returning the prediction given an input by an attacker. While the scenario is generally similar to many existing works, this paper aims at stealing the robustness of the victim model while maintaining the stolen model accuracy. The main idea is to find delta-uncertain examples to query the victim model to examine the boundary of the model. The experiment shows the approach outperforms baselines in terms of stolen model's accuracy and robustness.
strength_and_weaknesses: Strengths.
- S1: The paper proposes a new metric to evaluate a solution to an existing problem.
- S2: The proposed approach uses a reasonable threat model and approach.
- S3: The proposed approach is evaluated with various parameters such as victim model architectures, two datasets, without robustness training by the victim model, and 5 baselines.

Weaknesses.
- W1: The paper is unclear on how the pretrained model was used, and when it was used, especially for the baseline attacks. Since this is a very important factor based on the appendix, this needs to be clarified.
- W2: The baselines do not include Extraction-AT which is seemingly the best alternative, and this approach might be tuned to perform better. But instead, this is only shown to motivate the problem with a limited setting.
- W3: The main varying parameter for each configuration is epochs, not budget. This limits the understanding of alternative approaches and overall utility as the attacker is more limited by the budget than the number of epochs. Also, the maximum epoch tested here looks rather a bit too high as ResNet-18 on CIFAR can train much faster. Does the attack augment the data somehow with more epochs despite the limited budget?
clarity,_quality,_novelty_and_reproducibility: The paper is generally written clearly, and the threat model is clearly stated. The paper has novelty in assessing the robustness of a stolen model. The reproducibility is medium with some unclear parameters and settings such as how the pretrained model is used, what optimizer is used, and if the model is indeed trained on a single example in Line 12 in Algorithm 1.
summary_of_the_review: The paper is well written, and point out an interesting aspect of model stealing. The core of the approach sounds reasonable, although it is unclear if the conjecture is indeed true as the effect of delta-UE is only shown with the downstream task of model extraction. The paper also has many unclear points in the setting, such as the use of a pretrained model, and how the train part is performed. The baseline configurations are also unclear despite many experimental results in the appendix. The paper's performance is also based on the use of a pretrained model, which can have very high impact and such a model can carry innate robustness with a large pretrained data and being free from the adversarial example specific to the task. Although Jagielski et al. (2020a) briefly mentions the use of a pretrained model or available data on the web, it focuses on the data rather than the model. The model extraction has a lower impact for such a task with a pretrained model available, compared to many proprietary models such as credit evaluation model. Overall, this paper explores an interesting new metric to evaluate an existing problem, but it also has some unclear descriptions and limited impact lowering the rating.

-----
After reading the rebuttal and revision, I updated my recommendation to weak accept. I think the paper is now thoroughly prepared to be presented at a conference. The paper is generally well written, although a lot of contents had overflow into the appendix, and the comparison with many commonsense baselines is thorough. The problem is novel as well as the solution. My only concern to reserve a strong accept is the practicality as the robustness of a stolen model is usually afterthought, and the defender wants to prevent stealing in general, not specifically the robustness. As the ultimate goal of adversarial research is to understand threats and protect an asset from them, this might pose a limited impact. Also, the empirical result shows small improvement only despite beating the baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes how to extract the accuracy/fidelity of a victim model and its robustness (if this property holds for the victim model). First, it is shown that standard attacks like "Vanilla" and Knockoff Nets cannot extract the robustness property of the victim model. Second, extracting models with adversarial examples can transfer (to some extent ~ 20%) of the robustness of the victim model when the JBDA attack is used, however, it slightly decreases the clean accuracy. Third, combining the standard extraction followed by adversarial training on the attacker side can extract a good enough robust model but slightly decreases the accuracy. Finally, the authors propose a new approach (called BEST - Binary Entropy Searching Thief) that queries the victim model with uncertain examples. These examples are close to the decision boundaries, which help to extract robustness property along with the accuracy of the victim model. 
strength_and_weaknesses: Strength(s):

1. The paper shows many experimental results, however, it is not clear how the pre-trained model is used to initialize the stolen copy. 
2. The idea of extracting both functionality and robustness from a victim model is novel. 

Weaknesses:

1. There is no novelty in the proposed attack. This is obvious that the goal of an attacker is to extract the decision boundaries of the victim model. 
1. There is a  robustness gap between the victim model and its stolen copy.
2. The difference between the combination of standard extraction with adversarial training vs BEST is tiny (compare Figure 1c and 1d). 
3. The paper proposes only an attack and does not consider any possible mitigation/defense techniques against BEST.
4. The defense proposed in [1] slows down the attacks that generate uncertain queries (similar to UE) against the victim model. The defense makes the attack much more costly when the victim model is more uncertain about its predictions. Such a defense is very effective against BEST. A simple approach to check it would be to take the defense method even with the basic entropy cost (the information leakage of a given query is based on the entropy computed on the victim's softmax output for a given query) and then check how much slow down in the query time the defense would incur in comparison to attacking the undefended victim model. 

**References:**

[1] "Increasing the Cost of Model Extraction with Calibrated Proof of Work" Adam Dziedzic, Muhammad Ahmad Kaleem, Yu Shen Lu, Nicolas Papernot, ICLR (International Conference on Learning Representations) 2022.

clarity,_quality,_novelty_and_reproducibility: Comments:

1. Figure 1c - combining the standard extraction followed by adversarial training on the attacker side can extract a good enough robust model but slightly decreases the accuracy. The assumption to limit the query budget to 5000 is arbitrary. It should be shown how the attack performs with more queries against the victim model. 
2. Figure 1c and 1d - are the pre-trained models used in both cases as the starting point for the stolen copies?
3. Why is the performance of JBDA better than CloudLeak? (Figure 1b)
4. Attack cost - end of page 2. The other budget should be allocated for the training of the stolen copy. The adversary has to compare the total cost of extraction and the creation of the stolen copy with the cost of training a new model from scratch. 
5. Figure 2 illustrates intuitively that the UE (uncertain examples) are the most informative for the stolen copy since they capture most of the knowledge about the decision boundaries. 
6. Property \textbf{P1} is rather not clear. Figure 1b shows that JBDA potentially overfits too much to the adversarial examples since the clean accuracy is lower, but the connection to the robust feature is rather weak. 
7. The formulation of the double minimization problem in Equation 2 on page 5 is not clear. Why do you include the minimization of the loss value using the stolen/attack model $M_{\mathcal{A}}$? 
8. The results in Tables 2 and 3 are not useful since they do not compare with other methods but only present results for BEST. 
9. It should be mentioned (at least in the appendix) - how the ARD, IAD, and RSLAD work. 
10. According to ARD - "knowledge distillation using only natural images can preserve much of the teacher’s robustness to adversarial attacks (see Table 1), enabling the production of efficient robust models without the expensive cost of adversarial training." So, why doesn't the model stealing with clean samples extract the robustness of the victim model?
11. The results at the end of page 7 indicate that finding a simple uncertain example (using the generation of adversarial examples) is enough. This indicates that JBDA with FGSM might be further improved.
12. Note that standard vision and NLP tasks do offer pre-trained models. However, many real-world and specific tasks, for example, in the medical domain, do not have pre-trained models. 
13. "There can be other robust solutions, e.g., certified defense" How would you extract a model that was made robust via a certified defense?

Neat:

- Threat model: is there any service that exposes logits? For example, Google's API responds with confidence scores that are not even softmax scores. Other APIs (such as Clarifai), return only labels. 
- Figure 1 - it should be added that ResNet18 is used. 
- At the end of page 4: it should be - every sample in $\mathcal{R}^N$ and not in $\mathcal{R}^n$. 
- The quality of writing could be improved. 
- The main text merges with the caption of Figure 3. The authors should not modify the standard latex setting to that extent. 


summary_of_the_review: The paper shows how to extract a victim model with its accuracy and robustness. The experimental part is extensive. The insight is to use uncertain examples which lie on the junction of decision boundaries (not only adversarial which are usually on a decision boundary).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a method for model extraction attacks capable of stealing both robustness and functionality from the victim model. The proposed attack works by creating uncertain examples for querying the victim’s model. Most of the concerns the reviewers had in the initial reviews were clarified by the author response. While the method still has a big gap between the robustness of target model and the stolen model, reviewers agree it is a good first attempt in studying an attack for stealing a model from both robustness and clean accuracy perspective and may inspire future work in the direction. 
justification_for_why_not_higher_score: There is earlier work on crafting examples around the decision boundaries of a model so the method is not completely novel. The application is novel. The method doesn't work so well in stealing robustness properties of the target model. 
justification_for_why_not_lower_score: The paper is the first work to study the feasibility of stealing both robustness and clean functionality of a target model.
Decision: Accept: poster