Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel framework to trace the compromised model by only using a single adversarial example. The method can be applied in forensic investigation scenarios.
strength_and_weaknesses: The paper is well written. 

The application scenarios described in the paper are interesting and novel to me.

The experiment results show that their method is effective.

Some concerns are as follows:
1. What about the performance on large-scale datasets? For instance, ImageNet.
2. Can the proposed method be applied to other tasks? For instance, detection and segmentation?
clarity,_quality,_novelty_and_reproducibility: The paper is well written and the method is novel to me.
summary_of_the_review: The work is novel and interesting, but perhaps needs more experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a novel technique for detecting which model has been compromised (i.e. an adversarial example has been created targeting that specific model) from a set of deployed models that have been watermarked. Thus, each model deployed has a watermark so that some of the inputs are masked. When an attacker tries to compromise one of these models, the adversarial example contains some of the distinct features of the watermark introduced, which helps to detect the origin of the adversarial example. The authors consider two models: one where the defender knows the original and the adversarial example (data-limited adversary identification) and another one where the defender only knows the adversarial example (data-free adversary identification). The authors tested their proposed method on VGG16 and ResNet-18 models on CIFAR-10 and GTSRB datasets. 
strength_and_weaknesses: Strengths: 
+ The idea of trying to identify the model that has been compromised by an adversary using the proposed scheme with watermarking for forensic investigation is quite original and novel. 
+ The method is fairly intuitive and the paper is written and organized. 

Weaknesses: 
+ The paper lacks a more thorough analysis of the capabilities and limitations of the proposed scheme (see below) and to test the robustness against more advanced adaptive attacks (the one considered is a bit naïve). 
+ In the empirical results show that, in some cases, the data-free approach (with less knowledge for the defender) is less effective than the data-limited approach. This shows limitations on the proposed algorithms for the data-limited approach and suggest that the authors can easily improve the detection rate by considering also the score used for the data-free detection in equation (5). 
+ The watermarking technique is perhaps too simple, which I think that is limiting the performance of the proposed detection method.

clarity,_quality,_novelty_and_reproducibility: The idea proposed in the paper is quite original and novel and can help for forensic investigation when machine learning systems under attack. However, watermarking all the models with the proposed approach requires a significant effort form a computational perspective that may be limiting for deployments with many users/devices and large machine learning models. 
The intuition about the proposed method is clear. However, as I’ll explain below there are certain aspects that can be improved, both in the proposed method and the experimental evaluation.  

summary_of_the_review: Overall, I think that the idea explored by the authors is very interesting. However, the paper still requires more work:
+ The watermarking method proposed, as well as the detection techniques methods are simple and very intuitive. However, it looks like the adversary can also take advantage of this for evading detection. For instance, if the attacker is aware of the watermark scheme (not necessarily having access to the masks used to watermark the models), the attackers can also introduce bogus information in pixels/features where they think that are part of the mask used for the target model by looking at the perturbation introduced for each pixel. Alternatively, attackers can also play with the confidence level of the crafted adversarial examples, which may difficult the detection by the proposed algorithm. However, the adaptive attack used in the evaluation is too simple (it just adds Gaussian noise) and does not necessarily exploit the weaknesses of the proposed method. Thus, a more thorough analysis on the robustness against adaptive attackers is necessary. 
+ Related to the previous point, for instance, it is not clear to me that the proposed method can be robust against black box attacks generated with procedural noise functions or structure noise patterns, as the one proposed by Co et al. “ Procedural Noise Adversarial Examples for Black-Box Attacks on Deep Convolutional Networks” (CCS, 2019), where the masking of some pixels will be possible ignored by the attack. 
+ The watermarking scheme is possibly too simple, which limits the performance of the proposed method. It is also not very clear, how this can perform in typical computer vision datasets (e.g. ImageNet) with a lot more pixels. On the other hand, the detection performance for some of the attacks is not very good (for a single example). I believe this can be improved with a better watermarking scheme. 
+ As mentioned before, it is odd that the data-limited adversary identification algorithm performs worse than the data-free one in some cases. This suggests that the first algorithm can be improved (by for example using the score in equation (5)). On the other side, the combination of the different elements in the detection algorithms is not explored properly in the experiments. Thus, the parameters alpha and beta are cherry picked (or no explanation is given about how to select them). In this sense, a sensitivity analysis (ablation studies) would be necessary, as well as a mechanism to select these hyperparameters.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an adversarial identification technique in a multi-model owner-customer environment setup. The authors have incorporated the watermark-based technique to uniquely identify the network used in the generation of adversarial examples. The watermark somewhat needs to be present in the adversarial example for its unique identification. 
strength_and_weaknesses: + The idea of identifying the network vulnerable to adversarial attack is interesting and can be explored to develop defenses.

- The vulnerability of the technique against adaptive attacks is a major concern. In literature, several attacks have been developed to showcase the ineffectiveness of existing defense works; which only makes defense a challenging task.
- The limitation can also be seen from the progress of several effective transferable attacks and hence an attacker might not be utilizing one of the models the owner is sharing.


clarity,_quality,_novelty_and_reproducibility: The paper is easy to read.
The novelty of the proposed watermark technique is limited. The approach as described very close to the works related to pixel dropouts. 

[1] Cognitive Data Augmentation for Adversarial Defense via Pixel Masking, Pattern Recognition Letters (PRL), 2021
[2] Dropping pixels for adversarial robustness, IEEE CVPRW, (2019)

There is no mention of releasing the code to the community.
summary_of_the_review: While the idea of the paper is interesting; several limitations need attention:

1. The novelty as described in the section above is limited and needs justification along with comparison and contrast with the existing works. How many masks per network are generated and what is the impact of these masks? Are these watermark masks random, fixed, or optimized through the network?

2. Are these masks unique for each attack or need to be different for different attacks?

3. The assumption that an attacker will utilize one of the networks might not be ideal in the real world. Due to the tremendous success of black-box and transferable, it is hard to detect from which networks the images are generated if a different surrogate model or no model information has been used.

Crafting Adversarial Perturbations via Transformed Image Component Swapping," in IEEE Transactions on Image Processing, 2022

4. The success of tracing in Table 2 is close to random (50%) for most of the attacks, making the success of this approach questionable.

5. The adaptive attacks can seriously break the proposed defense. 

6. Limited evaluation of networks, datasets, and no SOTA attacks makes the proposed study a shallow work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose an identification mechanism that understands which model has been tested with adversarial examples, by leveraging a random-based watermarking technique that is kept inside the malicious points.
The methodology is tested in a black-box scenario, where the attacker can only retrive the score of a target classifier, but they can not access the weights or the data of the model.
Experiments are conducted in two ways: data-limited analysis refers to the usage of both the original sample and the adversarial one, while data-free analysis refers to tracing the compromised model with just one observation.
The authors show that the detection mechanism they implement, based on the scores of the pool of deployed models, is able to spot which one has been attacked in both cases.
The authors also test their attack in the presence of an adversary that tries to mimic the watermark by applying random gaussian noise.
strength_and_weaknesses: **Strengths**
* the paper presents an interesting forensics problem, that can be useful to understand if malicious actors are trying to attack a specific model.

**Weaknessess**
* **Uniqueness is stressed but not proved.** While all the methodology is built on the intuition that watermarking creates unique copies that help analysers to trace either the culprit or the asset under attack, the authors do not analyse if the resulting adversarial examples are really unique among all of the models they use. Such limits the contribution of the paper, and the best suggestion would be a better rephrasing that help the reader in understanding that uniqueness may not be guaranteed in this schema.

* **Data-limited analysis is unclear.** The authors state that there might be cases where both the clean sample and its adversarial version are available, by bringing an example on autonomous driving. While the example per se is not clear to understand, the assumption of having both sample is strong, and not supported in the paper. Also, this would similar to detect the adversarial example and the original point, which is not an easy task (and the authors did not comment on that). The authors need to discuss more why this case is realistic.

* **Adaptive attack is not adaptive.** The authors should test the attack when the defense is known, hence the masking is more accurate than just adding random gaussian noise. The provided experiment is needed, since it is good to show the effect of attack at different capabilities of the attacker, but also further experiments are needed. 

* **Wrong or unsupported technical statements.** The paper presents statements that undermine the contribution since they are either false or only partially true.
    * It is not true that adversarial examples are close to the boundary. If an attack is a minimum norm / minimum distance [a,b,c] then it is true. Otherwise, if the attack is computed with a maximum confidence attack (i.e. maximising the misclassification error), the sample can be found deep inside the target distribution [d, e]. Authors should check [f] for transferability of adversarial examples.
    * Equation 2 is confusing and not supported by any proof.
    * the paper says that "since adversarial examples should be identical to original examples visually, ground-truth label could be easily inferred" is not supported. As specified before, this problem is neither trivial or easy to formulate. If that would be the case, the authors should discuss in detail the statement.
    * the authors do not provide any support for the CE loss to be higher / lower when scoring an adversarial example computed on one model or another (Adversarial stability paragraph)


[a] Carlini, N., & Wagner, D. (2017, November). Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security (pp. 3-14).

[b] Pintor, M., Roli, F., Brendel, W., & Biggio, B. (2021). Fast minimum-norm adversarial attacks through adaptive norm constraints. Advances in Neural Information Processing Systems, 34, 20052-20062.

[c] Rony, J., Hafemann, L. G., Oliveira, L. S., Ayed, I. B., Sabourin, R., & Granger, E. (2019). Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4322-4330).

[d] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018, February). Towards Deep Learning Models Resistant to Adversarial Attacks. In International Conference on Learning Representations.

[e] Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., ... & Roli, F. (2013, September). Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases (pp. 387-402). Springer, Berlin, Heidelberg.

[f] Demontis, A., Melis, M., Pintor, M., Jagielski, M., Biggio, B., Oprea, A., ... & Roli, F. (2019). Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In 28th USENIX security symposium (USENIX security 19) (pp. 321-338).

clarity,_quality,_novelty_and_reproducibility: No code is attached to the paper for reproducibility.
The idea is nice and might be also novel, but it is not addressed properly.
summary_of_the_review: Technical contribution is not proved, part of the threat model can be better discussed to convey why it is realistic, experimental part regarding the adaptive attack is lacking, many technical flaws and unsupported statements.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, instead of enhancing adversarial robustness, the authors take the investigator's perspective and propose a new framework to trace the first compromised model by only using a single adversarial example in a forensic investigation manner. 
Overall, the reviewers recognize that the intuition is clear, the idea of identifying the network vulnerable to adversarial attack is interesting, the paper is well-organized and is to understand, and contains some novelty.
However, the reviewers raise several problems and concerns, including evaluation of adaptive attack, analysis, performance, ans so on.
During the rebuttal process, the reviewers still did not reach a consistent consensus of giving positive comments on this paper.
Besides, the most common comment is that this paper needs more work to make it complete.
Therefore, this paper is recommended to be rejected.
justification_for_why_not_higher_score: The reviewers raise several problems and concerns, including evaluation of adaptive attack, analysis, performance, ans so on.
During the rebuttal process, the reviewers still did not reach a consistent consensus of giving positive comments on this paper.
Besides, the most common comment is that this paper needs more work to make it complete.
Decision: Reject