Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigates the blackbox adversarial attacks against deep clustering models. It first proposes a blackbox adversarial attack using vanilla GAN. Then it evaluates the attacks against a number of SOTA deep clustering models and real-world images. It reveals that the proposed attack can significantly reduces the performance of these models, has minimal query complexity. Moreover, to further examine the robustness, it utilizes two unsupervised defense approaches, and observes unsuccessful defenses. And it also shows a successful attack on production-level API Face++.

strengths_and_weaknesses: Strengths:

This paper, as the authors claimed, proposed the first blackbox adversarial attack against deep clustering models. And it performed multiple experiments to show its success on various SOTA models, real-world datasets, query complexity, transferability, resistance to defenses, and production-level API Face++.

Weaknesses:

1. The blackbox adversarial attacks seem a straightforward application of vanilla GAN to this problem. Although the authors mentioned the challenges in building GAN-based blackbox attacks in Sec.3, we did not see any novelty or smart operations in Sec.4.1 to build up blackbox attacks. And the most challenging part of the direct formulation of attack objective has been bypassed by practically choosing the indirect, and hence the ratio of successful attacks among total raises concerns. 

2. The norm of adversarial noise, epsilon, may not be an appropriately interpretable one in experiments (Fig.3), since its reasonable threshold can vary significantly for different datasets. Obviously a large enough epsilon can always lead to a "successful" attack, while it may not be a meaningful one (it should be reasonably small such that this attack does not completely overwrite datasets). One has to formulate an index that quantifies the intensity of attacking noises over data.

questions: Besides the major questions in the "Weaknesses" part, there could be more deserving the authors' attentions:

3. The evidence of PCA might not be convincing. We assumes you perform PCA on the total datasets, including all classes. As a complicated clustering/classification task (not easily separable w.r.t. a few principle components), it is not supervising that all data mixed in PCA plot together with the adversarial ones (considering they are close to the original).

4. In Fig.6, why the "best" plot also has an errorbar. Did it show multiple "best"? How did you define the "best"? Maybe we missed something.

5. We failed to see the necessity or the supportiveness of the application of anomaly detection as a defense approach. Considering the adversarial attack samples are "close" to the original, it may be foreseen that they fail to be observed as anomalies.

6. The query complexity of the attack (Tbl.2) may not be a fair comparison. These attack mechanisms are different and the attacks are imposed at different stages of learning process. Not sure whether this comparison could tell something.

limitations: We do not see any potential negative societal impact of this paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an effective attack method against deep clustering models. Specifically, the threat model and attack objective are newly defined, reducing the attack to an optimization problem. Then, by combining a common GAN training loss with the attack objective, the authors design a GAN-based blackbox attack so that the resulting GAN generates adversarial perturbations for a given input image.

The power of the attack–effectiveness, query complexity, and transferability–is demonstrated from thorough experiments. Also, the authors tested the performance of their attack against two standard defense methods (robust deep clustering and deep learning based anomaly detection), showing successful attack results against them. Finally, they tested a blackbox attack against an existing face clustering service. Since the actual cluster membership is unknown in this case, the authors trained an open-source surrogate model and attacked via the transferability of the attack.

strengths_and_weaknesses: Originality: To the best of my knowledge, the paper contains novel ideas.

[[Strength]]
1. To the best of my knowledge, this is the first work about adversarial attacks against the deep clustering model.

Quality: 

[[Strength]]
1. The authors tested the attack against numerous clustering models and performance metrics. To explain, for the attack performance test, the authors prepare six different state-of-the-art models and three different clustering performance measures.
2. Moreover, the authors also explore various aspects of the attack–query complexity and transferability–to better evaluate the attack’s value.
3. The power of the attack is evaluated carefully against existing defense strategies.

[[Weakness]]
1. Visualization of the PCA patterns between adversarial examples and original samples may not be an effective way to analyze the reason.  A better visualization should also show the difference in the PCA components between detected adversarial examples, undetected adversarial examples, and benign original samples. (The last two sets of samples should show similar patterns, while the first set should show visually different patterns.)

Clarity: The paper writing is clear, and there is no issue with the clarity of the paper.

Significance: 

[[Strength]]
1. The power of attack looks to be very impressive. The attack significantly reduces the clustering performance for all 18 combinations (six different models, three different metrics).
2. The proposed attack seems to be effective against existing defenses. The experimental results may imply the need for a new defense method to ensure the robustness of deep clustering algorithms.
3. The authors experiment with the attack transferability and test it against an existing face clustering service (via an open-source surrogate model). This result shows an urgent threat of the attack in a practical setting.

questions: 1. While the paper demonstrated impressive performance against state-of-the-art deep clustering models, it would be interesting to attack some earlier deep clustering models. Since earlier models decompose the high-dimensional data to a low-dimensional representation, we can observe what happens in the low-dimensional space. (Those small perturbations in high dimensional may result in considerable differences in the low dimensional data.)
2. A follow-up suggestion from Question 1 is to perform anomaly detection over the low-dimensional representations of adversarial samples. To explain, prepare an earlier version of the deep clustering model as an embedding method. (Even though earlier versions have low performance in clustering tasks, it is okay because we will use it only to create low-dimensional representations) Then, create adversarial examples against a state-of-the-art model and embed those adversarial examples in low-dimensional space. Using any anomaly detection method, check whether the embedded representations are outliers or not.

limitations: The authors discussed limitations and potential negative societal impact in the paper appropriately.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a new black-box attack against deep clustering models. Specifically, the adversarial perturbation is computed by a generative model. It penetrates existing defense methods for clustering, and manifests transferrability which further make the proposed method practical. To further demonstrate the effectiveness of the proposed method, experiments are also conducted on a commercial API named Face++, apart from commonly used evaluation datasets in previous related works.
strengths_and_weaknesses: # Strengths

1. The manuscript focuses on run-time adversarial attack on deep clustering model, which is a less-explored area.

2. The proposed attack method is simple and intuitive. Experiments demonstrate its effectiveness.

# Weaknesses

1. [minor issue: paper organization] The introductory part of the manuscript can be improved. Since the proposed attack and defense methods are all based on deep learning, it is no longer necessary to introduce the clustering task starting from traditional methods like k-means. Instead, it is suggested to briefly mention (1) how a deep learning-based clustering method works, (2) how does attack against such method work, and (3) the potential risk induced by the attack to practical applications. These will make it more worthwhile for this topic to draw attention from the community (line 40).

2. [minor issue: unclear figure] In figure 1, it is unclear which picture correspond to which model mentioned on line 79 (SPICE, RUC).

3. [minor issue: potentially misleading] On line 35, it is pointed out that “zero-th order optimization is intractable for deep models.” in foot note. However, zero-th order optimization is a common approach for black-box attack against supervised deep classification models. See survey paper “Benchmarking Adversarial Robustness on Image Classification, CVPR2020” for examples, including ZOO, NES, as well as SPSA. In this sense, the word “intractable” is ambiguous, and may possibly convey wrong information to the reader.

4. [insufficient citations] This paper focuses on attack and defense problem on a specific domain other than classification. In section 2, the author should cite and discuss the main-stream attack defense works on the classification problem, and discuss how and why these ideas cannot be adapted for clustering.

5. [misleading contents] In the field of classification, “black-box attack” usually refers to run-time attacks against a trained and frozen model. See “Benchmarking Adversarial Robustness on Image Classification, CVPR2020” for examples. The attacks happend during training-time are called Backdoor attack or Poisoning attack. So please avoid using “adversarial attacks against clustering” as paragraph title on line 90. To be less ambiguous, it should be “Training-time attacks against ...”.

6. [important: relationship with classification attack] Based on the description on line 116. A frozen trained clustering model is a classification model to assign cluster labels from softmax probabilities. In this way the existing black-box (run-time) attacks should be compatible in the setting of this paper. However, comparison and discussion are missing. The aforementioned CVPR2020 survey paper can be used as a referece regarding this. Existing (black-box, run-time) classification attack methods can be used to attack the cluster assignment, e.g. based on the softmax scores. To be clear, score-based attacks like NES and SPSA in the mentioned CVPR2020 paper can do this. In this sense, the claim on line 138 is wrong. Score-based attack like NES and SPSA can optimize M (softmax scores) using estimated gradients even if the optimization target is completely indifferentiable.

7. [minor issue: unconvincing claim] On line 210, the ALRDC approach makes the latent space robust to perturbations. A robust latent space is already very important for adversarial robustness, and it not clear why this restricts its use. That said, evidence provided on line 212 indeed makes comparison with ALRDC unnecessary.

8. [confusing figure] Figure 2 shows an irregular pattern after attack. According to equation on line 148, the attack is untargeted, which means the objective does not care which would be the result class for misclassification. In this sense, shouldn't the confusion matrix be relative random instead of being concentrated on the “cat” or “horse” class?

9. [misleading title] The title mentions “defense”. But the paper actually only involves attack and evaluated this attack against existing methods for defense. The paper does not propose any new possibility for defense.
questions: My questions are written in weaknesses 4, 5, 6, and 8.
limitations: Limitations are not discussed in detail -- only several words in section 7.
Potential societal impact is discussed in the last paragraph. I agree that in order to be able to defend, we first have to know how to attack.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes the first black-box test-time adversarial attack against deep clustering models. This work helps bridge the adversarial analysis from traditional clustering approaches (explored before) to deep learning-based clustering models, where only white-box supervised learning based attacks have been explored. The attack uses Generative Adversarial Networks (GAN) with just score-based query access to the deep clustering model, with a more clearly defined threat model. The authors evaluate this attack on SOTA deep clustering models, in a transferability study, and on the Face++ ML-as-a-Service (MLaaS) system. The authors also demonstrate that existing defense approaches fail to stop the attack.
strengths_and_weaknesses: Strengths:
- Attacks a real-world MLaaS
- Evaluates against potential mitigation strategies
- Strong attack results

Weaknesses:
- Limited technical novelty
- Parts of the problem have been previously explored (e.g., traditional clustering, or white-box attacks on deep clustering), although the combination of factors is new
- Ambiguous epsilon bounds

questions: The paper is well-written and explores an interesting problem (the adversarial robustness of deep clustering techniques along with some mitigation strategies). Overall, I think this is a good submission, and am encouraged to see work on black-box attacks on real MLaaS systems.

One aspect that could be improved is in making further distinctions between the authors' attack and AdvGAN, as it seems that there are many technical similarities between AdvGAN and the GAN formulation in this work. Although it is adapted somewhat to the clustering problem, the overall formulation with an adversarial, hinge, and GAN loss is very similar to AdvGAN, limiting its technical novelty.

While generally well-written, there remain some things that could be further clarified in the paper. How is the surrogate to Face++ trained? Also, how are the epsilon bounds set for the table? It seems arbitrary given that the numbers in the Appendix are not consistent across datasets and it is unclear why they are different, so it seems some justification would help. Also in this case then, for the transferability studies, are the epsilons still unequal?

Do you have any insight into: 1) why there seems to be common specific patterns in the noise and 2) why most attacks are clustered in the same class?

Finally, I think more examples of specific real-world threat vectors could be included to better: 1) scope the societal impact of the work and 2) better motivate the need for robust deep clustering techniques.

Minor comments:
- Could clarify a bit what the specific threat model contribution is when brought up in the Intro (line 46)
- Intro: could cite some specific numbers on percentage of performance changes with the attack
- Could further distinguish between when a "score-based" black-box model is used vs. a "decision-based" black-box model

limitations: The authors have provided clear descriptions on the level of access needed to data, models, etc. to pull off the attack, and several mitigation approaches are mentioned and studied. The authors also discussed their hope that this work leads to successful development of defenses. Specific examples of real-world motivations and consequences of attacks on deep clustering algorithms could be further discussed.
review_title: null

Meta-review: metareview: To investigate the adversarial attacks and robustness for deep clustering models, the authors propose a blackbox attack using Generative Adversarial Networks (GANs) where the adversary does not know which deep clustering model is being used, but can query it for outputs.
Based on several rounds of discussions between the authors and reviewers, the reviewers' concerns have been properly addressed.
Since all reviewers consistently gave positive comments, the AC made a final decision of acceptance.



confidence: Certain
Decision: Accept