Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a simple but effective way to avoid model stealing. To the best of my knowledge, the idea is novel. The writing of this paper is quite clear and experimental results also show the effectiveness of proposed method. By introducing diversity loss, the model ensemble tends to produces discontinuous outputs which is hard to be distilled. I still have some questions:

1. In your experiments, there is an assumption that the thief knows what exactly the model archs. Given this prior knowledge, what will happen if the thief use a more complex/bigger model to distill ? 

2. It seems  inconsistent results are observed on Flowers17 dataset, any explanation?

3. There is another assumption that the attackers are constrained by attach budget (50000 trials), what will happen if more trials or unlimited trials are allowed? I think this question is important because this is often true scenario if the attacked models are of high business value.

4. Have you tried your method on larger tasks such as ImageNet or other fields, for example machine translation and speech recognition?
review_title: Effective Model Defense Technique, Need More Experimental Evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackles a timely problem of protecting deep neural networks from mode stealing attacks. This paper proposed an Ensemble of Diverse Model to provide diversity prediction for the adversary’s OOD query. The main contribution of this paper is the introduction of the diversity loss function on OOD data and the discontinuous prediction provided by ensembled models. The results show that EDM defense reduces the accuracy of stolen models.

Pros:
1.	The idea of using ensemble of diverse models to create discontinuous predictions on OOD datasets is interesting.
2.	The experimental evaluation is comprehensive. Five datasets and three attacks are investigated in the evaluation. Adaptive attacks using average predictions are considered in the evaluation as well.
3.	This paper is well-written. Figures make the paper easy to understand.

Cons:
1.	My major concern is the auxiliary OOD dataset used by EDM. Does the defender have access to the adversary’s OOD dataset? In the experiments, although the EDM used auxiliary OOD datasets, these auxiliary OOD datasets are more complex than the adversary’s OOD datasets. If the adversaries have access to the auxiliary OOD datasets or use other complex OOD datasets, will the defense still work?
2.	As mentioned in the paper, the hashing function requires transformation invariance, so that the adversaries cannot get the average prediction. However, can the DNN-based perceptual hashing function be generalized and applied to other types of transformation? For example, to break the transformation invariance, the adversary can use different algorithms to craft noises into the input data.
3.	From Table 3, it seems that the performance of EDM on JBDA and JDBA-TR is not very well. My understanding is that since JBDA and JDBA-TR do not reply on a fixed OOD dataset, EDM trained on auxiliary OOD dataset may be well generalized to these attacks’ OOD data. 
4.	The query budget of the adversary is set as 50,000. A larger number of queries are used in the previous attacks. For example, in Juuti et al. 2019, 102,400 queries are used to steal models trained on the MNIST dataset. Does the small query budget used in the paper limit the performance of attacks?
5.	Minor comments:
a.	What is CS in equation (1) and (2)? 
b.	“A perceptual hashing algorithm is used prevent adaptive attacks”
c.	“Our empirical evaluations shows that”

review_title: EDM's access to the adversary's OOD dataset

final_score: rating: 7: Good paper, accept
review_content: Summary: The paper proposes a method to protect deep neural networks against model stealing. The propose defense trains an ensemble of classifiers using two losses one targeting accuracy and the other diversity of the ensemble. In particular, the trained classifiers are consistent on in-distribution data, but contradict each other on out-of-distribution data. The proposed defense does not affect the test accuracy of the victim model while it strongly limits the test accuracy of the clone model.

Despite the concerns below, I vote for accepting the paper. The addressed issue is relevant, the proposed method is interesting. The performance in the presented experiments is convincing and the writing is overall reasonably clear.

Concerns:
1. the method hinges on keeping the hash private ("security through obscurity")
2. it also hinges on the attacker not being able to query the API with in-distribution images
3. the authors argue that the inference runtime is not impacted by the method, however, the GPU memory requirements scale linearly with the size of the ensemble, which might be costly

Questions:
- it seems one could try to launch an adversarial attack on the hash, why should it fail?
- how far from the in-distribution images would one have to move to start hitting the effect of the diversity loss? As a user of an API, I have some knowledge about which images are in-distribution, why is using this information impractical?
- at the end of Sec 4.1, results on generalization to unseen out-of-distribution datasets are referred to, supposedly to be found in Sec 5.3, but I cannot find them there, where are they?
- the hash function should map to as many members of the ensemble as uniform as possible on both the in-distribution and out-of-distribution data, how is this achieved?

Typos:
- Table 3, the number for hard-label - Flowers-17 - JBDA-TR - EDM should most probably be 16.54 not 6.54, please check

review_title: Good paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary
- The paper proposes a defense against recent flavours of model stealing attacks by exploiting the insight that the recent effective attack query out of distribution examples to the victim model.
- The approach introduces discontinuities in the input-prediction space by (i) training an ensemble of models such that for a given OOD input, the predictions are randomized (ii) at inference time hashing an input to a particular input in the diverse ensemble.
- Evaluation on simple datasets (MNIST, Fashion, CIFAR) show that the approach is reasonably effective at defending and moreover without a significant degradation of the utility.

## Strengths

**1. Well-motivated problem**
- The defenses for model stealing are not as well-investigated as attacks. I appreciate the authors take a step towards addressing defenses; the initial results look promising.

**2. Insight**
- I like the insight used by the authors in the defense i.e., to generate discontinuities in the input-prediction space. This is well illustrated in Fig. 1.

**3. Writing**
- The paper is written well and is easy to follow.

## Concerns

### Major Concerns

**1. $D_{in}$ vs. $D_{out}$**
- It is a nice insight that adversarial users indeed rely on querying data from a different distribution. However, my first concern is that is also naturally true for benign users. After all, no one except the victim/defender has access to the training data distribution and all queried data (whether by adversary or benign users) is out of distribution.
- As a result, I wonder how applicable the defense is -- would the defense intentionally mispredict in non-IID query setups? For instance, when queried with a particular dog breed unseen during training? Or querying slightly translated digit images to standard MNIST classifier?
- More generally, the proposed approach requires drawing a strict line between in and out distribution which I find is not clearly analyzed (follow-up remarks in point 4).

**2. Diversity Objective**
- If I understand the training objective correctly (Fig. 2b, Eq. 3), each model $f_i$ in the ensemble is encouraged to generate a random prediction when queried with out-of-distribution (OOD) inputs. Moreover, in expectation, results in a uniform distribution over predictions on OOD data. For in-distribution data, the predictions ideally remain unchanged.
- As a result, my concern is that this appears to be simple OOD detection followed by misprediction/calibration in disguise. Consequently, I wonder if one can simply leverage existing advances in OOD detection to achieve the objective.
- Moreover the OOD implicitly performed here requires access to a corresponding dataset $D_{out}$ to model unknown OOD samples. This contrasts some existing OOD detection works (e.g., ODIN, Liang et al., ICLR '18) which shows success without explicitly modelling OOD data.

**3. Hashing**
- I am also concerned that the hashing strategy (Sec. 4.2) employed by the authors appears to be a weak link in the defense.
- Specifically, it appears that a robust defense requires the hashing function to consistently select the same model in the ensemble in spite of small changes in the input. However, going by Fig. 6 it does not necessarily seem to be the case.
- As a result, I wonder if an attacker can exploit the hashing function to recover whether the output is clean/poisoned -- such as by aggregating predictions over a set of transformed inputs.

**4. Evaluation**
- While the results look promising, I have two concerns here related to the evaluation.
- First, I would have preferred if the results in Table 3 were presented as curves (with defense vs. attacker accuracy). This would have provided some insights on how the defense performs w.r.t the important $\lambda$ hyperparameter in their objective. Additionally, it would also make the comparison with AM baseline fair, especially given that AM was evaluated on a curve and it is unclear which specific instance was used for comparison.
- Second, which also connecting to point 1, is that the paper's experimental setting assumes a significant discrepancy between the distributions e.g., Victim's $D_{in}$ = MNIST, $D_{out}$ = KMNIST, Attacker's data = FashionMNIST. Here, the data and semantic classes are completely disjoint. I wonder how the defense performs with other choices of attacker's data e.g., EMNIST which is a bit more similar to MNIST.


### Minor Concerns

**5. Experimental settings**
- I find missing some important experimental parameters that is not mentioned in the paper: how many images were queried by the attacker for results in Table 1? What was the value of $\lambda$ used?

**6. Subverting schemes**
- I was also disappointed that the authors do not demonstrate that the defense is robust to some simple schemes used by an attacker to bypass the proposed defense. For instance, connecting to point (3), by aggregating predictions over a set of transformed inputs.


### Nitpicks

**7. Some nitpicks**
- $\lambda$ is overloaded: in Eq. 3 and also for the JBDA attacks
- "AM trades off benign accuracy for improved security" -- would the proposed approach also trade this off by increasing the value of $\lambda$?

### Post-rebuttal updates
- Thanks for the detailed response and additional evaluation.
- Q1. Fair point -- it appears that defenses do assume attackers query from a reasonably different distribution. This defense does make this assumption explicit by including it in the training objective ($D_{out}$ in Eq. 3). But then again, it seems typical for OOD-based defenses (e.g., AM).
- Q3. This concern also generally connects to Q6. It would be nonetheless interesting to analyze scenarios where the attacker attempts to use some auxiliary knowledge to break the defense. This is also shared by some other reviewers.
- Q4. Thanks for presenting the curves. Assuming strictly non-overlapping OOD data does seem like a strong assumption though.
- Nitpick editorial comment: Please serif-based fonts for text in equations e.g. $argmax(\cdot) \rightarrow \text{argmax}(\cdot), index \rightarrow \text{index}$
- I am slightly increasing my rating.
review_title: Blind Review

Meta-review: This paper proposed an ensemble of diverse models as a mechanism to protect models from theft. 
The idea is quite novel. There are some concerns regarding the robustness of the hashing function (that I share), however not every paper has to be perfect, especially when it introduces a novel setup. 

AC
Decision: Accept (Poster)