Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper raises a bar against an adversary who aims to reverse-engineer efficient models deployed to edge devices. A naive deployment practice allows the attacker to just extract the model's architecture and parameters which work as a prior for other attacks (such as adversarial attacks). To address this issue, the paper proposes ModelObfuscator that converts a victim model's architecture and parameters. It also obfuscates the ML frameworks by obfuscating the function names (corresponding to layers) and the ways they perform the layerwise computations. In evaluation, the paper shows that the resulting models return the same outputs on a set of input samples even after applying the obfuscation, and the obfuscation method does not incur large extra computations.
strength_and_weaknesses: Strengths:

1. The paper studies a research problem that may carry some importance in the future.
2. The paper proposes ModelObfuscator that employs six different ways to obfuscate a model.
3. The paper shows the proposed mechanism does not change a model's behaviors nor increase its computations.


Weaknesses:

1. The approach relies on "security by obscurity."
2. The evaluation against existing reconstruction attacks and adaptive attacks should be done.
clarity,_quality,_novelty_and_reproducibility: I like this paper as it studies a research problem that could be the problem soon. Reverse-engineering the models deployed to the edge may offer the white-box advantage to the attacker. It could be much easier to make other attacks on the model such as privacy attacks or adversarial attacks. Thus, the paper studies a defense.

However, there are three major problems of the proposed approach:

(1) Obfuscation cannot be a defense against reconstruction attacks. The paper cited several papers and insights from the software-engineering community, but those papers are unlikely to study code obfuscation as a defense. Rather, they study the obfuscation attacker to understand the limits of malware writers who aim to increase the time and effort of security analysts.

As a counter-example, the model extraction attacker who just queries the model to extract the parameters can still extract parameters precisely without reverse-engineering the model from the software stack.

Moreover, (let's assume that we exclude the extraction attacks) if the victim really wants to obfuscate the model parameters, one way is to encrypt them completely by utilizing TEEs. By doing so, the attacker who naively reconstructs the model parameters at the software level cannot get the actual parameters.

Thus, I recommend those changes:

> Clarify the threat model that the reconstruction attacker operates in.
> Tone down the entire paper; ModelObfuscator is not a defense or doesn't work against all reconstruction attacks.
> Clarify the goal of the mitigation mechanism.


(2) There should be an evaluation against reconstruction attacks. Prior work proposes three types of reconstruction attacks: (1) the reconstruction of model parameters via query, (2) the reverse-engineering of the model from the software stack, and (3) the reconstruction of model architecture from side-channel information. This paper claims that ModelObfuscator can be a defense against reconstruction attacks, but there is no evaluation against any of those adversaries.

Thus: 

> I'd like to see the results on the effectiveness of ModelObfuscator against those three attacks.


(3) There should be an evaluation against adaptive attacks. A conventional way to test the obfuscation mechanism is to assume that the attacker also has access to the same obfuscator (this is also used to evaluate the security of encryption mechanisms). Access to the obfuscator should not reduce the time and effort of the adversary who wants to reverse-engineer the victim model. 

Thus:

> I'd like to see the results of a scenario where the adversary can also use ModelObfuscator.
summary_of_the_review: This paper studies a research problem that can be important in the future (as we deploy more and more models to edge devices). To address this issue, the paper proposes a method that obfuscates the model internals (such as architectures, parameters, and computation details).

However, this defense has the following major problems:

(1) The defense heavily relies on "security by obscurity."
(2) The defense was not evaluated against any reconstruction attacks.
(3) The defense was not evaluated against an adaptive adversary.

Thus, I believe this paper (as-is) has more flaws than strengths. But, I am happy to update my review score if the authors address all the requested changes.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Motivated by the security of machine learning models running on device, this paper proposes ModelObfuscator, an obfuscation technique designed to protect models from multiple types of attacks, including adversarial examples, model extraction and membership inference. The proposed obfuscation includes five types of model changes, ranging from layer renaming, introduction of new layers and connections, and operation size masking. The correct predictions of the neural network are maintained after the obfuscation process. ModelObfuscator is implemented for TFLite models. Experiments are performed on ten TFLite architectures.
strength_and_weaknesses: Strengths:
- The paper proposes a method and implementation for obfuscating TFLite models.
- The experiments show that models that use ModelObfuscator cannot be parsed by model parsers.
- The implementation of ModelObfuscator is open-source.

Weaknesses:
- The motivation of the paper related to model security is loosely covered, inexact, and not followed throughout the paper.
- The experiments do not evaluate the proposed obfuscation w.r.t. the main motivation of the paper, security goals.
- More generally, the paper fails to motivate the need for neural network obfuscation, as neural network architectures are not particularly interpretable and sometimes even generated automatically via neural architecture search.
- The paper seems to lack scientific novelty and can be viewed, at best, as an engineering feat (trivial obfuscation strategies).

Please see details below.
clarity,_quality,_novelty_and_reproducibility: **Novelty**
- The obfuscation strategies proposed in the paper seem trivial.
- [Xu et al., 2018] seems closely related to the work in this paper, however it is not cited by the paper. How does ModelObfuscator compare to this prior work?

**Quality**

*Quality - related works:*
- Cited references on adversarial machine learning seem outdated, presenting defenses that have been proven ineffective as being state-of-the-art (e.g., adversarial example detection, cited as [Ma et al., 2018] in the paper).
- The related works section seems to only cite methods for producing adversarial examples in terms of attacks. This seems incomplete and skewed, considering that the paper is trying to encompass all types of attacks on devices.

*Security of machine learning:*
- The paper loosely tackles the notion of attacks against machine learning models on edge devices and claims that obfuscation is the solution to all of them. Instead, a threat model should be explicited, clearly stating what attacks are a risk on edge devices, which can be defended with obfuscation and it can be done. Moreover, the assumptions about the attacker and defender capacities and objectives should be stated.
- Multiple technical inaccuracies are present in the paper:
  * If the attacker has access to the model and forward pass, it is unclear how the defender can prevent differentiation. Most gradient obfuscation techniques can be defeated by gradient approximations. Moreover, none of the obfuscation techniques introduced in the paper seem to alter the gradients.
  * It is stated that model inversion is a white-box attack (Sec. 3.2). However, it should be clear that if the attacker has access to the model (which is what white-box means), they do not need model inversion anymore; they have the actual model. Model inversion only applies to black-box access, e.g., machine learning as a service (MLaaS) and prediction APIs.
  * Even if the attacker did not have white-box access to the model, model inversion most commonly has a goal to reproduce the functionality of a model in a surrogate. It is known that the actual architecture or hyperparameters do not matter, and the model can be leaked anyway with high fidelity (e.g., [Orekondy et al., 2020]). Based on the surrogate model, most other types of attacks can be performed (model inversion for intellectuall property loss, followed by adversarial examples and membership inference attacks). The paper does not address this aspect.
- Due to the lack of an explicit threat model, it it unclear what happens if the attacker is aware of the obfuscation strategy that is being used.
- The terminology used in the paper around machine learning security is not standard and not meaningful, e.g., trustworthy training, proactive defenses.

*Neural network obfuscation as security measure* Based on the points above, it is unclear how model obfuscation helps with any of the attacks used as motivation for the paper. It seems that the attacker has white-box access to the model on the device, which, obfuscated or not, is able to provide correct predictions. The attacker can use that model directly or build a simpler, non-obfuscated surrogate reproducing its functionality, that can be used for the attacker's end goal.

*Necessity for neural network obfuscation* Beyond the application of model obfuscation for security goals, I challenge the need for obfuscation in neural networks in a broader sense. As recognized in the paper, obfuscation started out applied to code and programming. There, the notion makes sense, as code is (mostly) written by humans and made to be understood by humans. Obfuscation makes code unreadable to humans. Or, neural network architectures are not particularly intuitive, and often enough the result of an automated optimization process. As such, the task of obfuscating them might not have many applications today.

*Quality of experiments:*
- Almost none of the experiments performed in the paper are related to the goal stated for the proposed obfuscation, which is the security of deployed models.
- A valid evaluation would be to craft attacks from each category that ModelObfuscator is trying to defend (e.g., adversarial example, membership inference).
- SmartAppAttack should be cited and briefly explained earlier in the paper.
- The paper does not provide a deeper analysis as to why model parsers fail on the obfuscated models. It might be for trivial reasons, such as the fact that these very recent parsers (2021-2022) are in preliminary stage of development, do not support custom layers, etc. It is also unclear what this experiment proves, as, for all practical purposes, the obfuscated models seem to still be functional.

**Clarity**
- The writing style needs improvements, as some sentences do not make sense.

**Reproducibility**
- The implementation of ModelObfuscator is open-source.

**References**
* [Xu et al., 2018] Hui Xu, Yuxin Su, Zirui Zhao, Yangfan Zhou, Michael R. Lyu, Irwin King. DeepObfuscation: Securing the Structure of Convolutional Neural Networks via Knowledge Distillation. 2018.
* [Orekondy et al., 2020] Tribhuvanesh Orekondy, Bernt Schiele, Mario Fritz. Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks. ICLR 2020.


summary_of_the_review: Paper of limited scientific contribution, with weak motivation and poor evaluation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose ModelObfuscator, a methodology that applies novel obfuscation techniques customised to hide relevant information regarding architecture and operations of the input neural network.
Among the obfuscation, ModelObfuscator can hide the real shape of the layers of the input neural network, re-arrange the data flow inside layers (but preserving the original one), and rename classes and functions.
The effectiveness and preciseness of the original model is preserved, with a negligible overhead.
strength_and_weaknesses: **Strenghts**

+ novel manipulations of neural networks that preserve the original performance, but rearranging and complicating the structure
+ the core contribution of the paper can help bringing the gap between software obfuscation techniques and AI technologies
+ converters can not deal with this new format of models
+ open-source code is released (but maybe the authors could have submitted a Dockerfile that would have make the testing easier)

**Weaknesses**

+ the authors do not show how their manipulations are more effective than regular Android obfuscation techniques. In particular, the code-related manipulations (such as the class renaming) are not novel. An example of previous implementations can be found in ObfuskAPK [a]. The authors might compare their techniques, showing that regular tools can not obfuscate part of the machine learning model.
+ converters do not work because this obfuscation is not known yet. Hence the authors should add this limitations to their work, or at least discuss what can be done once this obfuscation technique is known and how converters could be patched to take into account this technique.

[a] Aonzo, S., Georgiu, G. C., Verderame, L., & Merlo, A. (2020). Obfuscapk: An open-source black-box obfuscation tool for Android apps. SoftwareX, 11, 100403.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and details about obfuscation strategies are easy to understand.
The content of the paper is novel, and reproducibility is assured thanks to the inclusion of the code (anonymous link inside the paper)
summary_of_the_review: + Obfuscation techniques customised for hiding meaningful information of deployed model, such as architecture and layers' shape
+ Original performance is assured with negligible overhead
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work introduces ModelObfuscator, an obfuscation strategy for deep neural networks. ModelObfuscator introduces 5 model obfuscation strategies, namely renaming, parameter encapsulation, neural structure obfuscation, shortcut injection, and extra layer injection. The authors demonstrate that ModelObfuscator (a) prevents model extraction tools from parsing the deployed model, (b) obfuscated models incur a negligible time overhead with (c) a model overhead of around 20%.
strength_and_weaknesses: ## Strengths
(+) The topic of deep neural network obfuscation has not been studied extensively in the literature. This work holds novelty in the proposed model obfuscation techniques.  
(+) ModelObfuscator achieves convincing results, supporting the claims made by the paper.  
(+) The overall presentation of the paper is well-structured and easy to follow.  

## Weaknesses
(-) While the authors claim that their proactive defense could disallow a transformation from a non-differentiable model to a differentiable one, in a realistic attack scenario an attacker could simply train a differentiable substitute model by querying the deployed model to create his own dataset. It seems ModelObfusctator cannot prevent this simple attack. It would also be beneficial if the authors could elaborate on their claim why ModelObfuscator could disallow a transformation from a non-differentiable model to a differentiable one, and in which scenarios it is possible and in which it is not.  
(-) In essence, the shortcut injection and extra layer injection are decoys to confuse the attacker. However, under the hood, the model architecture is not changed since the library will ignore these parts. It would be more interesting if such an obfuscation could be achieved by really changing the model architecture with the proposed methods.  
(-) The authors did not include related work on model obfuscation [A-F]. Consequently, the authors failed to differentiate themselves from the existing literature and to compare their approach against existing methods.  
(-) Given, that this work introduces obfuscation tricks, which are not really changing the underlying model architecture to make it more secure, I am not sure if this work is in the scope of the topics for ICLR. This work might be more suitable for a conference on the topic of security & privacy or software engineering.  

[A] Anonymizing Machine Learning Models; ESORICS 2021  
[B] DeepObfuscation: Securing the Structure of Convolutional Neural Networks via Knowledge Distillation; ArXiv 2018  
[C] Mimosanet: An unrobust neural network preventing model stealing; CVPR-W 2019  
[D] Preventing Neural Network Weight Stealing via Network Obfuscation; Intelligent Computing. SAI 2020   
[E] Hardware-Assisted Intellectual Property Protection of Deep Learning Models; ACM/IEEE Design Automation Conference (DAC), 2020  
[F] Preventing DNN Model IP Theft via Hardware Obfuscation; EEE Journal on Emerging and Selected Topics in Circuits and Systems 2021  

clarity,_quality,_novelty_and_reproducibility: ## Clarity
This work is clearly written and can be easily understood by the reader.

## Quality 
The quality of the paper is good. 

## Novelty
This work holds some novelty since the topic of model obfuscation is not studied extensively in the community. 

## Reproducibility
The authors provided their source code and relevant information in the paper. To the best of my judgment, this work will be reproducible. 

summary_of_the_review: I believe that this work holds some novelty. However, given the insufficient related work and my concern if this work will be of interest to the ICLR community I am currently leaning toward rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents prototype tool, termed ModelObfuscator, to automatically obfuscate on-device TFLite models. Experiments show that ModelObfuscator increasing the overhead of extracting information about the models (such as structure, parameters and attributes), without increasing latency.

Reviewers raised several concerns:

-The approach relies on "security by obscurity", which is strongly discouraged by many security experts, since an adversary with enough time and resources will be able to extract the valuable information eventually.

-Experiments are not conducted with state of the art model extraction attacks.

-An attacker could simply train a surrogate model, via "model theft", to circumvent the proposed obfuscation strategy.This would result in a black-box attack which is true and would indeed be more costly. However, this does not change the fact that this model would be able to be attacked and extracted, which would finally render the proposed defense useless.


justification_for_why_not_higher_score: Please see above
justification_for_why_not_lower_score: NA
Decision: Reject