Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: HODA is a new defense against model extraction attacks. It is a detection method that determines if a given user is adversarial based on the hardness of users' queries. For a given sample, the hardness metric is computed as the epoch number of a subclassifier after which the sample is classified correctly. The method generates histograms for in-distribution data and the histograms of each user's queries. The Person distance is computed between the histograms and if the distance is greater than a certain threshold $\delta$ then the user is classified as malicious. The results show that the hardness degree histogram is an indicator of dataset distribution. In general, the in-distribution samples have significantly lower hardness levels than samples used by adversaries (the adversarial samples are OOD natural or synthetic data).
main_review: **Strong points:**
1. The method outperforms previous detection-based defense called PRADA.
2. The method is well-described but some more intuition why it works well would be helpful. Overall, the paper is well-written.

**Weak points:**
1. The storage overhead is substantial if the method has to store checkpoints after every epoch. However, it is shown that this requirement can be alleviated by storing checkpoints only every 10th epoch and it is sufficient to maintain a high detection rate as well as the low false-positive rate of the HODA defense. Additionally, defenses usually come at a cost, so this can be justified. The authors could compare to PRADA not only in terms of storing the hardness histograms but also add the size of the saved checkpoints.
2. The computational cost of the method is increased by a number X of saved checkpoints in comparison to the standard inference. Usually, the DNN models are large and if only a single K80 GPU is available, then HODA cannot be executed in parallel (not sufficient memory). Additionally, the GPU load + energy cost is increased by X as well.

**Other points:**
1. There are other detection-based defenses, e.g. [1] and [2].
2. Minor inconsistent naming convention, e.g.: CIFAR10 and Cifar10.
3. Why is MobileNet performing the best across the 3 models in Figure 2?
4. Many transfer learning methods freeze all but the last (classification) layer. How does HODA perform in this setting (or only if few last layers are retrained)? What is the dependence between how many last layers are re-trained and the detection rate of HODA?
5. In Appendix G, PGD examples are not hard with hardness degree **<** 70.

**References**

[1] Stateful Detection of Model Extraction Attacks. Soham Pal, Yash Gupta, Aditya Kanade, Shirish Shevade. https://arxiv.org/abs/2107.05166

[2] Extraction of Complex DNN Models: Real Threat or Boogeyman? Buse Gul Atli, Sebastian Szyller, Mika Juuti, Samuel Marchal, N. Asokan. https://arxiv.org/abs/1910.05429

summary_of_the_review: The paper presents a new detection-based defense against model extraction attacks that performs better than the previous (directly comparable) defense called PRADA. However, the defense is expensive in terms of computation and storage usage.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: In this paper, the authors propose HODA (hardness oriented detection approach) to detect model extraction attacks against DNNs. They use the checkpoints during training of the model to define hardness of a sample. Hardness of a sample is the epoch from which the model assigns the same class to the sample. During inference, Pearson distance between the histograms of holdout set samples and test samples is used to detect an attack. HODA is evaluated on image datasets and models. It is shown to detect three classes of attacks and is analyzed against an adaptive attack.

main_review: The authors use the ordered ensemble of checkpoints to give a numerical hardness value to image samples and argue that the distribution of benign test samples by hardness differs from attack samples. Using this, distance is computed between holdout set histogram and test time histogram. If it exceeds a threshold, an attack is detected.

The definition of hardness appears to have conceptual flaws:
* It does not account for mis-classified samples. A mis-classified sample $x$ should intuitively be one of the most hard samples and should be assigned the highest hardness degree $m-1$ (where $m-1$ is the last epoch). Among the epoch-wise checkpoints, $x$ could get a wrong label $l$ in some epoch $e < m-1$ and the model could continue to label it as $l$ in the later epochs. So the hardness degree $\phi_{f_t}(x) = e < m-1$, which is counter-intuitive.
* Let $A$ be a model that is not well-trained (e.g., trained using a very small learning rate) and the original labels of the training samples do not change at all. Then all the samples will get hardness degree of $0$! In comparison, a well-trained model $B$ will have samples falling into higher values of hardness. Thus, by the hardness definition in the paper, the well-trained model $B$ would find the same set of samples "harder" than an ill-trained model. There does not seem to be any correlation between model accuracy on a dataset and hardness of the dataset.
* There is no guarantee in general that model training is monotonic for each example, that is worst to better. So concluding hardness based on the latest epoch number at which its label stabilizes is not indicative of its hardness.

Because of the conceptual flaws, I am not convinced about validity of the proposed approach.

There are several limitations of the experiments:
* The paper compares only against PRADA, which is not designed to work for attacks based on natural samples (e.g., Knockoff nets used in the paper). There are at least two defenses which are designed to detect such attacks: Atli et al. "Extraction of Complex DNN Models: Real
Threat or Boogeyman?" and Pal et al. "Stateful detection of model extraction attacks". The authors should compare against these defenses. The paper incorrectly states that PRADA is the only defense in this space.
* To construct benign users, the authors take a split from the original test distribution. In practice, a benign user can provide samples that could come from similar distributions and not necessarily from the same distribution. The authors should conduct experiments to check false positives for such cases, e.g., MNIST vs color MNIST.
* The results on adaptive attacks show that HODA requires an order of magnitude more samples (50-100 to 500-1K) to detect a decent number of attacks. For the CIFAR10 model, it requires even more samples. HODA requires running each sample on multiple checkpoints, which can be expensive in the adaptive case due to increasing number of samples required for detection.
* Appendix H gives trend over different hardness degrees for the target and extracted models. This however does not tell us the model accuracy numbers exactly. The authors should report model accuracy for these models and for models extracted when HODA is applied.

summary_of_the_review: The paper is targeting a challenging problem. However, the central concept of hardness of samples as defined in the paper appears flawed. The paper has several limitations when in comes to experimental evaluation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: - The paper proposes a defense 'HODA' against DNN model stealing attacks.
- They key idea of the approach is a strategy that discriminates a set of 'benign' examples (i.e., similar to victim's data distribution) vs. 'surrogate' examples (i.e., from attacker's input distribution). 
- The strategy involves mapping an example $x_i$ (benign/adversarial) to 'hardness' value which roughly indicates at which point in the training process the intermediate model converges on prediction of $x_i$. HODA exploits a property that the hardness distribution statistics varies between benign and surrogate sets.
- Experiments on standard datasets {CIFAR10, CIFAR100, CUB200, ...} and attacks {knockoff, JBDA, ...} indicates that HODA is able to achieve reasonable detection rates and FPR against attacks.
main_review: **Strengths**

1\. Novelty and certain benefits
- While there has been some recent attention to defenses via manipulation, detecting queries targeted for model stealing has receive little attention. The paper explores the latter line and demonstrates empirical improvements.
- The defense also provides some benefits e.g., defender does not need attacker's model architecture or parameters, it can be combined with manipulation-based defense strategies.

2\. Experiments
- I appreciate the authors comparing against multiple recent attack strategies and also evaluating in a variety of settings.

3\. Well-written
- The paper is well-written and was easy to follow.

**Major Concerns**

1\. Constraints / Severe assumptions
- Although the results appear promising, I am concerned that these results are a product of possibly impractical assumptions and would appreciate the authors' clarification:
- (a) examples can be precisely associated with a user $u$: what if the user $u$ performs a sybil attack and spreads the examples among multiple different user accounts? (which seems easily achievable in MLaaS scenarios.) 
- (b) to obtain hardness values of a example now requires 10-100$\times$ as many predictions (to classify over all $f^m$ classifiers)
- (c) distribution(benign set) = distribution (training set): but this seems highly unlikely as benign distributions can have a very specific dataset bias (e.g., objects with white-backgrounds, images under poor lighting). I would guess in this case the benign user would be considered adversaries? 

2\. Hardness analysis and comparison to OOD
- A key contribution of the paper is the concept of 'hardness' that captures some statistics of a set of examples and contrasts it with a normal set consisting of the victim's training examples. While the results using hardness appear promising, I find missing some additional analysis or discussion justifying 'hardness'. After all, the defense task is reduced to comparing two distributions and this is achievable by a variety of techniques (e.g., FIDs, Wasserstein, MMD). While the authors show that hardness is a good measure, I find lacking *why* or some in-depth analysis motivating hardness.
- I appreciate the empirical study in Sec. 4.3. that shows hardness captures OODness of samples. This makes me wonder if there are connections to existing OOD detection approaches (e.g., ODIN ICLR '18) and if one could instead simply use OOD detection techniques (which has significantly progressed recently) to defend against model stealing attacks. 

3\. Experiments / Evaluation
- OOD baselines: Building on top of my previous concern, I would have appreciated comparison of HODA with OOD detection approaches (e.g., ODIN) or similar OOD techniques used within model stealing literature (e.g., Adaptive misinformation, Kariyappa and Qureshi CVPR '19). These seem more competitive and fairer to compare HODA with. Moreover, this seem more appealing from a defense perspective, since they typically perform example-level classification (vs. user-level) and as a result only certain examples can be rejected.
- Table 3-4: To claim that  HODA outperforms PRADA in Table 3-4, I find overall results missing e.g., FPR vs. detection rate as a result of modifying $\delta$, AUROC numbers. Going by the table alone, I am unsure if there exists a range of thresholds $\delta$ where HODA performs poorly.
- Simulating benign users: How are benign users $S_u$ simulated -- by using a held-out fraction of the victim's test set? If yes, this seems concerning as it is possible to design the approach or tune the hyperparameters and overfit to biases in the victim's distribution. One idea would be to construct the benign sample set by using images from CIFAR10 classes, but sampled from TinyImage or another dataset.


**Minor Concerns**

1\. Large hardness values -> noisy?
- For histograms in Fig. 1,3 wouldn't the values at higher hardness values be extremely noisy? For instance for a hardness value of 98 (at m=100 epochs), this can be achieved using only predicts at a single reference epoch (e=99). This seems to be case going by the curves where there is a big peak at higher hardness values. As a result, it might make sense to train the model instead of 200 epochs and estimate hardness values until 100.
summary_of_the_review: - The proposed approach 'HODA' extends the state of defenses against model stealing attacks by exploiting an interesting property of how an attacker's distribution contrasts a benign distribution. The experimental results indicate that indeed this property can be exploited to some extent to reject malicious queries.
- However, I am primarily concerned about (a) certain assumptions (being able to associate each example with the same user, distribution comparison) and (b) how this compares to simply performing OOD detection. In particular with (b), if the key idea to exploit that attacker's distribution is semantically different from the victim's, I would have appreciate a better comparison with OOD detection, or present some analysis/insights of the proposed 'hardness' justifying the choice.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Hardness-Oriented Detection Approach (HODA) to detect the sample sequences of model extraction attacks.

main_review: This paper demonstrates that the hardness degree of samples is important in trustworthy machine learning. The authors investigated the hardness degree of samples and demonstrated that the hardness degree histogram of model extraction attack samples is different from the hardness degree histogram of normal samples.

My comments are as follows.

First, in section 4.1, this paper states a definition of the hardness degree, which is essential in this paper. However, I didn't see any significant result from Figure 2. Also, the meaning of accuracy in the Y-axis is unclear. In the last paragraph of page 4, it is not lucid to me how the authors get the results based on this figure.

Second, in section 4.2, the authors introduce the model extraction attacks and evaluate them. Nevertheless, I am confused about how to get the attack budget and how to get these images. I think the authors should add more details about that.

Next, in section 4.3, the authors calculate the hardness degree of the samples generated by attacks. However, I am wondering for the data samples that have a large hardness degree, is it easy to distinguish them from the data samples not used for span queries. As mentioned in the next section, the Pearson Distance will be very large. So, I don't think we can use the datasets that contain plenty of such samples to measure the sequences.

Finally, in section 5, the authors proposed the experiment setup and evaluation. However, I would like to know more evaluation metrics in the table, not only FPR.

Minor: in the last paragraph of page 7, it should FPR not PFR.

summary_of_the_review: In general, I think this paper has merit.
review_title: null

Meta-review: The paper presents a new method for detection of model extraction attacks. It is based on the intuition that typical model extraction attacks involve samples submitted by users that are harder to classify than "benign" samples submitted by users. By introducing the notion of hardness, a metric is developed for identifying malicious users submitting their samples for the purpose of model extraction. While the proposed method is original, it incurs a substantial overhead. Experimental evaluation of the proposed method also has several deficiencies, in particular, in the assessment of its overhead as well as in modeling of benign users.
Decision: Reject