On the Robustness of Deep Clustering Models: Adversarial Attacks and Defenses
Authors: Anshuman Chhabra
Ashwin Sekhari
Prasant Mohapatra
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Clustering
Adversarial Attacks
Visual Learning
Robust Learning

Abstract: Clustering models constitute a class of unsupervised machine learning methods which are used in a number of application pipelines, and play a vital role in modern data science. With recent advancements in deep learning-- deep clustering models have emerged as the current state-of-the-art over traditional clustering approaches, especially for high-dimensional image datasets. While traditional clustering approaches have been analyzed from a robustness perspective, no prior work has investigated adversarial attacks and robustness for deep clustering models in a principled manner. To bridge this gap, we propose a blackbox attack using Generative Adversarial Networks (GANs) where the adversary does not know which deep clustering model is being used, but can query it for outputs. We analyze our attack against multiple state-of-the-art deep clustering models and real-world datasets, and find that it is highly successful. We then employ some natural unsupervised defense approaches, but find that these are unable to mitigate our attack. Finally, we attack Face++, a production-level face clustering API service, and find that we can significantly reduce its performance as well. Through this work, we thus aim to motivate the need for truly robust deep clustering models.

On the Robustness of Deep Clustering Models: Adversarial Attacks and Defenses
Anshuman Chhabra", Ashwin Sekhari", and Prasant Mohapatra
Department of Computer Science University of California Davis, CA 95616
{chhabra, hatpaar...eeetrt.oooopppaaaaaaar
Abstract
Clustering models constitute class of unsupervised machine learning methods which are used in a number of application pipelines, and play a vital role modern data science. With recent advancements in deep learning- deep clustering models have emerged as the current state-of-the-ar over traditional clustering approaches, especially for high-dimensional image datasets. While traditional clustering approaches have been analyzed from robustness perspective, no prior work has investigated adversarial attacks and robustness for deep clustering models in a principled manner. To bridge this gap, we propose a blackbox attack using Generative Adversarial Networks (GANs) where the adversary does not know which deep clustering model is being used, but can query it for outputs. We analyze attack against multiple ofte-ooftthe-rr deep clustering models and real- world datasets, and find that it is highly successful. We then employ some natural unsupervised defense approaches, but find that these unable mitigate our attack. Finally, we attack Face+t, a production-level face clustering API service, and find that can significantly reduce its performance well. Through this work, we thus aim to motivate the need for truly robust deep clustering models.
1 Introduction
Clustering models are utilized in many data-driven applications to group similar samples together, and dissimilar samples separately. They constitute powerful class of unsupervised Machine Learning (ML) models which can be employed in many cases where labels for data samples are either hard (or impossible) to obtain. Note that there are multitude of different approaches to accomplishing the aforementioned clustering task, and an important differentiation can made between traditional and deep clustering approaches.
Traditional clustering generally aims minimize a clustering objective function defined using a given distance metric [. These include approaches such as k-means (2). DBSCAN (3]. spectral methods among others. Such models generally fail to perform satisfactorily on high-dimensional data (i.e., image datasets), or incur huge computational costs that make the problem intractable [5]. To improve upon these approaches, initial deep clustering models sought to decompose the high-dimensiooal data to cluster-friendly low-dimensions representation using deep neural networks. Clustering was then undertaken on this latent space representation (6] 77. Since then, deep clustering models have become considerably advanced, with state-of-the -he-tthrrt models outperforming traditional clustering models by significant margins on number of real-wur sal-world datasets {8] 9..
Despite these successes, deep clustering models have not been sufficiently analyzed from robustness perspective. Recently, traditional clustering models have been shown to be vulnerable adversarial
"Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

attacks that can reduce clustering performance significantly [10 However, no such adversarial attacks exist for deep clustering methods. Furthermore, no work investigates generalized blackbox attacks, where the adversary has zero knowledge of the deep clustering model being used. This is the most realistic setting under which a malicious adversary could aim to disrupt the working of these models. While there is a multitude of work in this domain for supervised learning models, deep clustering approaches have not received the same attention from the community.
The closest work to ours proposes robust deep clustering 9 12 by retraining models with adver- sarially perturbed inputs to improve clustering performance. However, this line of work has many shortcomings: 1) it lacks fundamental analysis on attacks specific to deep clustering models (for e.g., the saate-of-hhe-nr robust deep clustering model RUC [9] only considers images perturbed via FGSM/BIM [13 [4] attacks, which are common attack approaches for supervised learning). 2) no clearly defined threat models for the adversary are proposed and 3) there is no transferability [15] analysis. Thus in this work we seek to bridge this gap by proposing generalized blackbox attacks that operate at the input space under a well-defined adversarial threat model. We also conduct empirical analysis to observe how effectively adversarial samples transfer between different clustering models.
We utilize Generative Adversarial Net- works (GANs) [16] for our attack, in- spired by previous approaches (Adv-
Original
GAN [17. AdvGAN+ [18), etc) for supervised learning. We also utilize number of defense approaches (es- sentially deep learning based anomaly detection [19] and dfae-of-the-aa "ro-
pre-allack
dluster
dog
cleet
hone
dog
ship
Advensorial
poat-allack
clister
ship
plane
6ng
dog
hidd
bust" deep clustering models [9] to
Figure I: Adversarial samples generated by our attack (first 4 image pairs from the left correspond to SPICE and the others to RUC).
determine if our adversarial samples can be mitigated by an informed de-
fender. One of the major findings of our work is that these approaches are unable to prevent our adversarial attacks. Through our work, we seek to promote the development of better defenses for adversarial attacks against deep clustering.
Finally, to truly showease how powerful our attacks can be, we attack a production-level ML-as-a- Service (MLaaS) API that performs clustering task (and possibly utilizes deep clustering models in the backend). We find that our attack can also significantly reduce the functioning of such MLaaS clustering API services. To summarize, we make the following contributions:
We propose the first blackbox adversarial attack against deep clustering models. We show that our attacks can significantly reduce the performance of these models while requiring minimal number of queries. We also undertake a transferability analysis to demonstrate the magnitude of our attack.
We undertake thorough experimental analysis of most taatte-o-thh-ane (SOTA) deep clustering models, on a number of real-world datasets, such as CIFAR- (0). CIFAR- 100 [21]. and STL-10 which shows that our attack is applicable across number of models and datasets.
22
We show that existing (unsupervised) defense approaches (such as anomaly detection and robustness via adversarial retraining) cannot thwart our attack samples, thus prompting the need for better defense approaches for adversarial attacks against deep clustering models.
We also attack a production-level MLaaS clustering API to showcase the extent of our attack. We find that our attack is highly successful for this real-world task, underscoring the need for making deep clustering models truly robust.
Figure shows some of the adversarial samples generated by our attack for the SPICE S and RUC [9] deep clustering models on the 1TL-10 dataset, and their corresponding pre- pre-attack and post-attack predicted cluster/class label To the human eye, these samples appear indistinguishable from each
"These attacks cannot he used for deep clustering hecause: 1) they employ computationally intensive optimizers that use exhaustive search and hence, make the attack too expensive for high-dimensional data, 2) they are designed for traditional clustering and are training- time attacks, but, deep clustering models are deployed frozen for inference, requiring test-time attack.
'For example, for RUC [9]. it implicitly assumed that the adversary has knowledge the dataset as well as ground truth labels, and will attack using supervised whitchox attacks.
"Class labels are inferred by taking the majority from the ground truth labels for samples in that cluster.
2

---

other, but the model clusters them incorrectly after the attack. As we will show in later sections, our attack on SPICE for STL-10 results in a 83.9% (67.9% for RUC) decrease in clustering utility measured according to the NMI (3 metri
2 Related Works
Deep Clustering. To leverage clustering algorithms ligennimnsi data, early work on deep clustering 6 7. aimed to learn a latent low-dimensional cluster-friendll representation that could
then be clustered on. This task achieved using AutoEncoders (AEs)
7 24 25 Variational
AEs [26 27]. or GANs [2 30). More recently, current SOTA deep clustering models employ
self-supervised and contrastive learning instead of the earlier AE/GAN based approaches to perform
clustering 8 9 31.. We consider these in the paper due to their superlative performance.
Training-time Adversarial Attacks Against Clustering, Recently, few works have proposed blackbox adversarial attacks against clustering [T]. These seek to poison a small number of samples in the input data, so that when clustering is undertaken
the poisoned dataset, other unperturbed samples change cluster memberships. This problem is significantly different from ours- attack at
the traditional clustering algorithm retrains on the poisoned input data constituting training time. For deep clustering, the model does not train again once deployed, so we have to generate adversarial images that the model misclusters test time. There have also been whitebox attacks proposed for traditional clustering, but these can only be employed for the specific traditional clustering algorithm considered 32-34
Attacks/Defenses in Supervised Learning. The closest attacks to ours in supervised learning are score-based blackbox attacks where the adversary has access to the softmax probabilities outputted by the model [35], A number of such attacks have been proposed: NES [36), SPSA [37], among others. These cannot be applied in their original formulation as the attack optimization expects ground truth reference labels. Even with simple modifications to the loss (such as using predicted labels as ground truth) these attacks do not work successfully our proposed GAN attack (refer to Appendix [.]] for a detailed discussion and Appendix [E] for empirical justification). Similar issues make supervised defenses inapplicable for our setting- defense strategies need be truly unsupervised to work in the clustering context, and outputs of deep clustering models might not possess a one-to-one mapping with ground truth labels, causing further problems. We provide more detailed discussion on the inapplicability of these defense methods in Appendix F22
Robust Deep Clustering. A natural in in-processing defense approach to preventing adversarial attacks on deep neural networks is to utilize some form of adversarial retraining which seeks to jointly optimize adversarial objective along with the original loss function used to train the network. This technique has been shown to vastly improve the adversarial robustness of traditional deep neural networks 38 39]. In the context of deep clustering, to the best of our knowledge, there two works that aim to make models robust to adversarial noise in this manner RUC [9] and ALRDC [2]. Moreover, RUC is considered be the SOTA robust deep clustering model. Even in terms of performance metrics (NMI, ACC, ARI) it is second only to the SOTA model SPICE 4
3 Preliminaries and Notation
In this section, we introduce notation and preliminary knowledge regarding the clustering task and the proposed adversarial attack. Note that for matrix of size u x v, we can index into row as A, Vi E [u]. and index into a single value at row i and column j as Aij, Vi E [u].] E [e]. Moreover, ||-I| denotes the Euclidean (2-norm) norm of a vector.
Deep Clustering. Since we are proposing blackbox attacks in our paper, we will be defining deep clustering models in more generalized manner that abstracts their inner functioning. We defer the reader to [5] for more details on the models analyzed in the paper. A deep clustering model is denoted C and operates on samples of the given dataset X consisting
samples and maps them
Siiiilar results for SPICE hold for CIFAR- 100 with 78.9% (58.2% for RUC) decrease in NMI, and for CIFAR-IO 10 with 72.9% (68.7% for RUC) decrease in NMI.
40 22 refer to "robustness" in their work but their definition is not the traditional notion of adversarial robustness and they do not incorporate adversarial retraining.
3

---

to one of k clusters. As deep clustering models utilize deep neural networks internally, they generate a set of softmax probabilities indicating cluster memberships, denoted as M E [0, 1| From this, the cluster labels can be obtained by taking the maximum of the cluster probabilities. For a dataset sample X; E X, the cluster label can be obtained as I argmax JE(A] [Maj]. We denote the vector of cluster labels computed in this manner as L E N
Note that unlike supervised learning problems, the ground truth labels Y N are not utilized for training the model in deep clustering. However, these ground truth labels are used for evaluating the performance of the model. That is, the output cluster labels L are evaluated in comparison with the ground truth Y. Performance metrics such as Normalized Mutual Information (NMI) [ Adjusted Rand Index (ARI) [44), and Unsupervised Accuracy (ACC) [45 are commonly used in deep clustering literature for this purpose. These metrics are defined analytically in Appendix A.6
Threat Model. We now define the threat model for the adversary:
1. The attacker has knowledge of the dataset X. This is a commonly made assumption in adversarial attacks against traditional clustering literature [T0
2. The attacker carries out a blackbox attack and does not know which deep clustering model is being used, but can query it to observe M (softmax cluster memberships), and hence, 7 In adversarial attacks on (un)supervised models, this is often what constitutes a blackbox attack 446
3. The goal of the attack is provide minimally perturbed images (there is noise threshold that cannot be exceeded) as input to the deep clustering model. Upon doing so, the model should miscluster these samples and the performance measured via the evaluation metries (ACC, NMI, ARI, etc) should significantly reduce post the attack.
Ad Adversary's 's Objective. The goal of the attacker is to input adversarial images to the model and lead to performance decrease, measured using metries such as NMI, ACC, and ARI. We cannot directly optimize post-attack M or L as the adversary does possess ground truth labels. Moreover, cluster labels generated by the model may not be the same as the ground truth labels, thus requiring some notion of a mapping function, further complicating the problem. Instead we will indirectly achieve this goal through another objective function.
Let an input image be X; E X and through single query the cluster probabilities Mj can be obtained before the attack. To attack, the adversary will introduce a carefully crafted perturbation/noise specific to this sample. The attacker queries the deep clustering model and obtains the set of cluster probabilities for this sample. Abusing notation slightly, these are denoted as C(X, + +d)
Assuming the original unperturbed cluster labels/probabilttis accurately depict the cluster represent- ing the sample's 's ground truth label, the attacker can simply generate the adversarial noise via the following optimization problem:
max
I| C(X: + +6]l s.t. ||8|| S
(Attacker's Optimization)
Here the constraint with c is simply to ensure that the adversarial sample does not have unbounded noise and remains realistic to a human observer. The objective function above ensures that the cluster probabilities post the attack C(X, + 5) are as distinct as possible to the cluster probabilities M; obtained before the attack. Then, assuming that prior to the attack the cluster labels were the correct representative of the ground truth label 8 the deep clustering model will miscluster the sample after the attack and performance (NMI, ACC, ART) should reduce. If the adversary introduces many such adversarial samples, the performance can thus be significantly affected
'I only L can be observed it is a decision- based attack; if M can be observed it is a score-based attack [35] 'This is true for most samples in the dataset clustering performance of the model prior the attack is assumed to superlative, otherwise there is no reason to attack.
'Note that this is an untargeted attack. One could also consider targeted attacks by replacing M, in the objective with length vector where the target cluster entry is 1. and all other entries are 0.

---

4 Attacking Deep Clustering Models
4.1 Blackbox Attack Using GANs
We utilize a simple GAN based architecture for generating the adversarial perturbation ó and solving the attack problem delineated in the previous section. There are many different variations to using GANs for generating adversarial samples, such as AdvGAN [I7], AdyGAN+ [18]. WPAdvGAN (5). CycleAdvGAN (26,, among many others. For our attack, we employ a vanilla GAN architecture consisting of deep neural networks for both the Generator and Discriminator, similar to AdvGAN.
We utilize the Generator model & to generate the adversarial perturbation ō for given input image X; E X, i.e., ((x)) + 8. The Discriminator model 'D plays similar role as for the original GAN model 16 as aims ensure that the perturbed image similar to the distribution of input images.
We then rewrite our attack optimization problem in the context of the GAN architecture. The loss for the attack objective can be written directly as in the optimization problem:
Lattakk == Ex, M c(x; + (X)))
And we can simply reformulate the constraint on the norm of the adversarial noise as:
Ceonstraint := Ex, min(c ||9(x))11.0)
We also write the vanilla minimax GAN loss [6 as:
L = E. (log(D(X.)) log(1 D(X: + ((X)))
To train the Generator 9 and Discriminator D, we then optimize these combined losses by solving the following saddle-point problem (where Oa, Og are hyperparameters to control tradeoff):
max min { -aaaattck Gc
G
Upon obtaining the trained Generator G. we can generate the adversarial perturbation as = (X,,) for any image X, E X provided as input. We then provide these adversarial images X; + 5 as imput to the pre-trained deep clustering model C to obtain cluster membership confidence scores as M/ C(X; + 5) after the attack. From M, we know the original cluster label L,; argmax M and similarly, we can obtain the cluster label of the adversarial image as L argmax Mi.."
If the optimization problem solved successfully and the distance between M, and C(X; + 6) is sufficiently large enough, we can have T42 * T.. Moreover, assuming that a mapping function ø exists that maps the output cluster labels to the ground truth labeling, we know that: ((T.)) Y,- Since LA * L4 we can conclude that Φi [.)) * Y, leading to misclustering and a drop in performance.
Remark. Note that since our attack objective is indirectly formulated, it is possible that M, and M2 are not sufficiently far apart even after the attack, leading to L; = L.G. However, as we find and our experiments show, this does not happen frequently in practice our blackbox attack is highly successful at generating adversarial samples that disrupt the performance of all deep clustering models considered in the paper (Section 6.. We have considered the following open-poureeeeeee deep clustering models in experiments: CC 31I. SPICE BJ, SCAN [T]. MICE [52), NNM [33, RUC
4.2 Attacking Face++: A Production-level MLaaS Clustering API
To showcase the disruptive capability of our attacks, we attack Face+ which provides an extremely well perforeefo face clustering REST API service. At a high-level, the API performs a basic face clustering task where it takes in input face dataset and seeks to cluster images belonging the same person together. We do not know the clustering algorithn/moder being utilized in the backend. Note that there is one significant difference here compared to the threat model for open-source deep clustering models previously considered- we do not have access to cluster memberships, but just the final set of labels. To overcome this problem, we attack Face++ by training surrogate open-source deep clustering model on the dataset and then generate adversarial samples for this model using our
htttps: http://wwwwaacolppppuusus.. wttp:/t.cccooo//eeeeee http//phooooallonnnllsttttto..ee clustering/
5

---

GAN based attack pipeline. We then use these adversarial samples as imput to the Face++ API to conduct the attack. Hence, our attack constitutes transferabilityy attack via a surrogate model.
The Face++ API service functions as follows: we first create a new face album using the createAlbum endpoint which gives wwccalbum_ttoeeeeeeeo token response. Using this token, add the images for our dataset using the addimage endpoint, and then call the groupF ace endpoint to per- form the clustering task. Finally, we obtain cluster labels for each image using the getAlbumDetail endpoint, which returns proup_ids as the integer cluster labels. We provide the results of our attack and experiment details in Section 6.3 as well as additional implementation details in Appendix C
5 Mitigating the Attack: Possible Defense Approaches
Robust Deep Clustering, Adversarially retrained models (where the learning process incorporates a joint adversarial loss along with the original loss) have been shown to considerably mitigate adversarial attacks, and thus constitute a natural in-processing defense approach. In the context of deep clustering, RUC and ALRDC utilize adversarial learning improve model robustness. However, we only consider RUC in this paper for the following reasons: 1) RUC is an add-on module, and can be applied to any deep clustering model, 2) in contrast, the ALRDC approach specifically works only to make the latent space robust to perturbations (which most deep clustering models considered in our paper do not possess), restricting its use, and 3) unlike RUC, from our early experiments with ALRDC we found that it did not work well with high-dimensional real-world image datasets such as
In-processing Defense:
LOCCIFAR-I00/STL-II
which we have considered in this paper
In our experiments in Section 6.2 we show that using our GAN based attack (Section 4.1 and RUC the deep clustering model C, can disrupt its performance significantly as well. Once the generator has learnt how to generate adversarial noise specific to RUC, it can easily degrade RUC's clustering ability. These results clearly indicate that there is deficiency in current robust deep clustering strategies, and the problem requires different solutions [12]
Deep Learning Based Anomaly Detection. With advancements in deep learning, the long-standing field of anomaly/outlier detection has also seen major advancements. Further, due to the unsupervised nature of the deep clustering task (no labels), anomaly detection is suitable pre-processing approach for detecting adversarial samples. In particular, most deep learning based anomaly detection models are trained specific datasets and can detect ofitt--diisrriutioon samples with extremely high precision. For our experiments, we use a recently proposed self- supervised deep anomaly detection approach SSD [99 which achieves state-of-the-ee performance on benchmarks when labeled training data is not present [56].
Pre-processing Defense:
While detailed description of SSD is beyond the scope of our work, the authors employ contrastive self-supervised representation learning combined with a Mahalanobis distance based threshold detector in the feature space to detect anomalies. In our experiments in Section 6.2 we show that even SSD is unable to detect a large majority of our adversarial samples. We further show that this is likely due to the distribution of our adversarial samples in space, they mimic the original samples fairly well due to the norm constraint on the generator's 's output. We use Principal Component Analysis (PCA) (57) to analyze the adversarial and benign samples and find that their principal components tend to be very similar. In this regard, anomaly detection approaches are also unable to detect most of our generated adversarial samples (less than 1% in the best case, and z 14% average!).
Experiments and Results
6.1 Blackbox Attack Against Deep Clustering Models
For our experiments on the open-source models, utilize the following real-worls image datasets: CIFAR- 10 (20), CIFAR- 100 (1], and STL-10 (22). These are commonly utilized in a majority of deep clustering literature. For the models, we consider a number of ofttee-arrrr-rrtttt deep clustering models: We also consider RUC (. but discuss
SPICE BJ, SCAN [5T) NNM [33, MiCE [52). and CC [3I]
II the ALRDC paper only MNIST 54 and Fashion-MNIST 55 were considered in their experiments.
12One simple but compute-intensive intensive solution could be to jointly minimize the deep clustering loss on adversarial inputs by using our trained adversarial noise generators, similar to traditional adversarial retraining.

---

its results as part of the defense Section 6.2 Notably, SPICE and RUC (add-on to SCAN) are SOTA #1 and #2 for performance on the aforementioned datasets. We show large number of adversarial images generated by our attack model and provide additional experiment details in Appendix A
Table 1: Pre-attack and post- aost-attack performance for open-source deep clustering models.
Model
CIFAR-I
10
CIFAR-
100
STL-10
NMI
ACC
ARI
KMI
ARI
ACC
CC
Post attack
Pre-attack
0.01
0.70
0.64
0.00
0.10
0.79
0.03
114.3
0.27
0.00
0.07
0.43
0.03
0.76
0.73
0.00
0.85
MICE
Pre-attack
0.73
0.69
0.8.5
11455
0.29
0,44
0.56
0.46
0.62
Post-attack
0.20
0.44
0.15
0.04
0.20
0.30
0.20
0.43
NNM
Pre-attack
0.75
0.71
0.84
0.48
0.32
0.48
0.70
0.65
0.81
Post-attack
0.30
0.09
0.41
0.18
0.01
0.15
0.22
0.07
0.25
SCAN
Pre-attack
0.71
0.66
0.82
0.49
0.33
0.51
0.67
0.62
0.79
Post-attack
0.27
0.06
0.48
0.10
0.01
0.14
0.16
0.02
0.22
SPICE
Post-attack
Pre-attack
0.23
0.85
0.10
0.84
0.36
0.92
0.12
0.57
0.39
0.02
0.15
0.54
0.14
0.87
0.87
0.00
0.16
0.94
Pre-attack
0.83
0.81
0.90
0.55
0.39
0.33
0.78
0.74
0.87
RUC
Post-ant Datt-kttack
0.26
0.08
0.33
0.23
0.05
0.26
0.25
0.07
0.30
As part of our experiment, we generate adversarial images (adversarial set) using our attack for all the images in the original test set. We show the performance metrics (ACC, NMI, ARI), pre-attack (original images) and post-attack (adversarial images) for all the aforementioned models and datasets in Table I Quite evidently, it can be observed that performance for all models is significantly reduced post the attack. Note that since the GAN network generates fixed noise for the same input, there is no variance in the obtained results. We have presented hyperparameter (aa, Dee etc.) values for all experiments in this section in Appendix A. due to space limitations.
Effects of the Attack. For all the deep clustering models, we find a general trend emerges in how the models' post-attack clusters differ from the clusters generated on the original test set. particular, performance the benign images is generally very good- this can be observed in the confusion matrix for the SPICE model and STI.110 dataset in Fig 2a Note that most of the clusters are correctly defined and there are a few miclusterings. However, for the adversarial images, we notice that there a complete clustering breakdown- most images tend to get lumped in a small number of select clusters, and the remaining few images create small-sized clusters. This can also be seen in the confusion matrix in Fig 26 for SPICE on adversarial STL-10 images. Most of the adversarial images clustered part of the "cat" cluster, despite being clustered accurately before the attack. This trend is prevalent for the attack on all the models, and tends to be more drastic for less performant deep clustering models, such as CC. This can be seen in the confusion matrices shown in Figs 22 and 2d We present the confusion matrices for the other models in Appendix A.3 due to limited space.
airplene-
sirplene bind cat-
airplane
bird
car
cat-
cat- dear
cat
dear
o00r
deer
horse-
horse-
monkey
monkey-
monkey
monkey-
ship
sh
sNi-
ship-
truck
trck
arplons
bley
Car
devs
harten
stap thack
prrplnn
car
brick
Irpllan
cal
mooiig
brunk
aippann
bldg
cat
slop
tort chip
trick
(a) Pre-atlack (SPICE)
(b) Ptst- (SPICE)
(c) Pre-atlack (CC)
(d) Post-atlack (CC)
Figure 2: Confusion matrices showcasing the effect of the attack for the SPICE/CC models on STL- 10 dataset.
A major takeaway from these results is that while deep clustering models tend to work very well on "clean" data, simple adversarial samples exist that can completely degrade performance. It is thus imper- ative that deep clustering model designers evaluate their models against adversarial samples.
Table Query complexity of the attack. Model
CIFAR-10
CIFAR-100
STL-10
CC
5148
5382
2150
MiCE
2820
3142
2244
NNM
2320
113360
4660)
SCAN
3100
2200
3080
RUC
3500
4000
3380
SPICE
2320
7520
2304
Query Complexity and Perturbation Analysis. We also measure query complexity of our approach to analyze the cost associated with carrying out our attack in the real world. In this blackbox attack
scenario, the query complexity is defined the number of times the deep clustering model is queried
7

---

by the adversary. Thus, we can measure the number of times the Generator ς queries the clustering model C with an input batch, before the loss of the GAN network converges. We present these results in Table 2 where the batch size is 256. It can be seen that the query complexity of our attack is quite minimal for all models and datasets. particular, the values obtained for our method are comparable to the query complexity rates obtained by existing blackbox attacks against supervised learning models [46 48). Moreover, a smaller query complexity is doubly beneficial in our case, because once the generator has been trained, we can use it to generate the optimal perturbation for any images that belong to that input distribution without requiring any additional queries to the clustering model.
1.0
1.0
1.0
0.3
CC
0.9
CC
0.9
CC
0.8
SCAN
0.8
SCAN
0.8
SCAN SPACE RUC
0.7
SPACE
0.7
SPICE
0.7
0.5
BUC
0.5
RUC
0.5
2 0.5
MMM
8 0.05
RNM
0.5
KNM
3.4
VICE
0.4
HIZE
0.4
HICE
0.3
0.3
0.3
0.2
0.2
0.2
0.3
0.2
0.1
0.0
30
20
20
40
30
70
03
90
100
0.0
10
20
30
40
50
60
70
00
90
100
0.0
15
20
30
40
50
50
70
90 100
Perturbation
Perturbation
Perturtation Norm
(a) NMI
(b) ARI
(c) ACC
Figure 3: Performance as a function of the adversarial perturbation norm for STI-10.
We also analyze the effect of varying the noise penalty (via c) the extent to which the attack degrades the performance of the deep clustering model. We depict the NMI, ARI, ACC for all the models the STL-10 dataset can be observed that as the norm threshold is increased the attack becomes more successful and the performance of the models is worsened, eventually plateauing close to O. To conserve space, we defer these results with similar trends for the other datasets to Appendix A..4
function of the norm of the generated perturbation in Fig 3 It
Transferability Analysis. We undertake transfcrability analysis to see whether adversarial samples generated for one model transfer to other deep clustering models. We present these results Fig 4 for all our datasets as transferability matrices. In these, we show the post-attack NMI for source and target models, along with the pre-attack NMI for each of the deep clustering models. It can be seen that the overall transferability of most adversarial samples is high. Note that SPICE (SOTA #1) is in general a much better deep clustering model than CC in terms of performance. Interestingly however, can that SPICE's adversarial samples do not transfer well to CC and vice versa. It would be interesting to investigate the reasons for this occurrence in future theoretical or empirical work. Due to space limitations, we show the transferability matrices with ACC and ART in Appendix A.5
S2AM.
SCAN
SCAN
CCAM
SCAN
3PICE
9234
SPICE
SPICE
SPICC
SPICE
C:
VICE
Eaijet
16:34 S2AV
SPICE
Pre-ttick
VICE
1argel Model's
sPice
RUC
Ppo sttcktckkk
PICE 10:M SCAK
Karpel Madds
sPict RUC
Pre
(a) CIFAR-10
(b) CIFAR-100
(e) STL-10
Figure 4: Transferability results showcasing post-attack (and pre-attack) NMI for different source/target models.
6.2 Defense Approaches
Robust Deep Clustering, As mentioned before, we utilize the RUC add-on module as first defense against our attack. As in the original paper, we utilize RUC with SCAN the deep clustering model and then carry out our GAN based attack against it. Similar to the experiments on the open-source models in Section 6.1 we measure the performance on a benign test set and then on a corresponding adversarial set generated using our GAN attack. We find that our attack is also successful against RUC, thus proving that RUC is not entirely robust to adversarial noise. This is shown in Table [ It can be seen that RUC is not satisfactory defense approach for our attack. However, note that the robustness of RUC is somewhat observable in our perturbation norm versus performance analysis shown in Fig 3 In general, it takes higher norm thresholds to reduce the performance of RUC, although it is ac achievable without adding too much noise to the images. For the transferability analysis shown in Fig we find RUC behaves similar SPICE, RUC's adversarial samples are highly transferable across all models and datasets except for CC on the CIFAR-10 10 and CIFAR- 100 datasets.
8

---

6000
Adversarial
Original
5000
OFAAI 011)
6000
O7AR300 10g2
S1 10 INLI
3000
OPARAO (DNL))
OTAAJ00 (5W0)
SILID
MICE
NMM
SPICE
Vedas
(a) SSD, SOTA deep learning based anomaly detection ap- proach unable detect a large majority of our attack samples (Here, Injected + Inj. and Delected + Det.).
(b) Visualizing the PCA components original samples (blue) and their adversurial counterpars (red) computed for the SPICE model. Figure 5: Results on using anomaly detection approaches as a possible defense.
Deep Learning Based Anomaly Detection. We now present results for SSD, the deep learning based anomaly detection approach, as a pre-processing defense to our attack. In our experiments, we start with test set of 8000 images for all of our datasets. For each benign image X,, we uniformly ran- domly either add adversarial noise to it or remain benign. We do this over 100 trials and average results observe how many adversarial samples were detected by the anomaly detection approach. We present these results in Fig Sal We find that on average the method detects 1..99, 19.41 %, and 11.71%, of adversarial samples for the CIFAR-10, CIFAR-100, and STL- datasets, respectively. However, this is negligible compared to the large number of adversarial samples being injected and remaining undetected. To analyze the reasons for this occurrence, we compute the first 3 principal components of the adversarial set and the benign set using PCA and present these in Fig 56 for SPICE, Visually, the principal components of a large majority of adversarial samples are superimposed and interspersed amongst those of the benign samples, showcasing the potency of our attack.
6.3 Attacking Face++ (MLaaS API)
We now present our results for attacking Face+t, a production- level face clustering API service. Since the service only works with face images, we use the Extended Yale Face B [58) dataset for these experiments. There and 500 images for each person. However, due to rate limits and the latency associated with uploading images using REST APIs, we test the service by sending only 10 images per person, for a total of 280 images. Then, only cluster labels are outputted by the API, we resort to using a surrogale model for the attack via transferabil- ity. We train a CC model as the surrogate on the entire Extended Yale Face B dataset and then train our GAN network to generate the adversarial noise for given image from the dataset. Thus, we generate adversarial counterparts the original 280 benign images and carry out the attack. Subsequently, we observe the NMI/ACC/ARI pre-attack and post-attack.
0.8
28 persons in this dataset, i.e., 28
0.6
Fre Artack Best Mttck
0.4
Figure 6: NMI/ACC/ARI before and after the attack on Face++ APL.
Since the results obtained be affected by which 10 representative images were picked for
each person in the test set, we randomize this selection process and take the average over 10 runs. As can be tack, the Face++ clustering API boasts stellar performance on the test set, with average NMI zo 0.97. However, for the set of adversarial im- ages, performance of the API service degrades significantly, with the NMI dropping down to 0.62 on average. This shows that our gener- ated adversarial examples can disrupt the work- ing of high-performance real-world service, even when we cannot query the model for memberships. We also show some of the adversarial samples generated from original images, and how they have been misclustered in Fig 7 Before the
Original pro attock
B22
B12
B21
B33
B33
8:39
in Fig 6 before the at-
Adversarial
post-allack.
Beca cherler
B12
B18
B18
B18
B12
B12
B18
Legend
Figure 7: Adversarial samples for the Face++ attack.
attack, most of the faces are clustered correctly, but after the attack they only majorly belong to the

---

B12 or B18 face clusters. Here too, we are observing the clustering breakdown effect previously noted for the attack against open-source deep clustering models. We provide additional implementation details and empirical results for this attack in Appendix
7 Conclusion
In this paper, propose the first blackbox adversarial attack against deep clustering models, using simple GAN based architecture (Section[4.1). We evaluate our attack against a number of open-source SOTA deep clustering models and real-world image datasets, and find that it significantly reduces the performance of these models, while having minimal query complexity (Section 6.J]. To further examine the robustness of deep clustering networks, we utilize two unsupervised defense approaches: adversarially trained "robust" deep clustering models, and deep learning based anomaly detection. We note that even these approaches are unable detect our adversarial samples and mitigate the attack (Section 6.2). Finally, we use surrogate model to attack production- levell face clustering API Face++ proving that our attacks are also effective against real-world MLaaS models (Section 6.3).
Broader Impact. Our findings are important as they underscore the need to making deep clustering models robust to real-world blackbox adversaries. As show, current robustness/defense approaches are unsatisfactory, and allow an informed adversary to reduce the performance models nonetheless. Another takeaway is that robust deep clustering model designers should evaluate their models against adversarial samples crafted specifically for deep clustering models (and not those crafted for supervised models, as in (2). In the wrong hands our attack approach can have negative societal impacts, but we believe through this work we can actually drive the development of truly robust clustering models, thus offsetting the impact of such threats. We positive that improved defenses and robust models will arise as a result of this work. Finally, to further detail the possible impact of our adversarial attacks, we provide practical attack scenarios in Appendix H that utilize our attack setting and threat model.
Limitations. There are few limitations of our work well: 1) For the most effective attack, all the conditions for the threat model need to be met. However, as we show for the Face++ API, even not met (for Face++ softmax cluster memberships could not be obtained),
when some conditions our attack can be successful. 2) Parameterization of the GAN for the attack is a non-trivial problem (for example, choosing e). This limits its applicability as it might be challenging attack model resources are limited and efficient hyperparameter search cannot be performed. 3) Our attack assumes knowledge of the training dataset used for the model, which might not be realizable in a number of attack scenarios. To overcome this, as part of future work, we believe partial information attacks can be explored against deep clustering models. 4) Our attack requires training GAN, deep learning model, which is a computationally expensive task. It might be worthwhile to explore better optimization strategies that can utilize our Lansek loss to disrupt the deep clustering function while using less computational resources.
References
[1] Lior Rokach and Oded Maimon. Clustering Methods. In Data Mining and Knowledge Discovery Handbook, pages 321-352. Springer, 2005.
[2] Stuart Lloyd. Least Squares Quantization in PCM. IEEE Transactions on Information Theory, 28(2):129-113 137, 1982.
[3] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In KDD, volume 96, pages 1996.
[4] Andrew Ng, Michael Jordan, and Yair Weiss. On Spectral Clustering: Analysis and an Algorithm. Advances in Neural Information Processing Systems, 14, 2001.
[5] Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang. Jianjing Cui, and Jun Long. A Survey of Clustering with Deep Learning: From the Perspective of Network Architecture. IEEE Access, 6:39501-39514 2018
[6] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised Deep Embedding for Clustering Analysis. In ICML, pages 478-487. PMLR, 2016.
10

---

[7] Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards <-means-friendl Spaces: Simultancous Deep Learning and Clustering. In ICML, pages 3861-3870. PMLR. 2017.
[8] Chuang Niu, Hongming Shan, and Ge Wang. SPICE: Semantic Pscudo-labeling for Image Clustering. arXiv preprint 0rX1v22103.99382, 2021.
[9] Sungwon Park, Sungwon Han, Sundong Kim, Danu Kim, Sungkyu Park, Seunghoon Hong. and Meeyoung Cha. Improving Unsupervised Image Clustering with Robust Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12278- 12287, 2021.
[10] Antonio Emanuele Cinà, Alessandro Toreinovich, and Marcello Pelillo. Black-box Adversar- ial Attack for Poisoning Clustering. Pattern Recognition, 122:108306, 2022.
[1]] Anshuman Chhabra, Abhishek Roy, and Prasant Mohapatra. Suspicion free Adversarial Attacks on Clustering Algorithms. AAAI, volume 34, pages 3625-3632, 2020.
[12] Xu Yang. Cheng Deng, Kun Wei, Junchi Yan, and Wei Liu. Adversarial Learning for Robust Deep Clustering. Advances in Neural Information Processing Systems, 33:9098-9100 9108, 2020.
[13] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adver- sarial Examples. arXiv preprint arXiv: 1412.657, 2014.
[14] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial Examples in the Physical World. In Artificial Intelligence Safety and Security, pages 99-112. Chapman and Hall/CRC, 2018.
[15] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in Machine Learn- ing: From Phenomena to Black-box Attacks using Adversarial Samples. arXiv preprint arXiv:1605.07277, 2016.
[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. Advances in Neural Information Processing Systems, 27, 2014.
[17] Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating Adversarial Examples with Adversarial Networks. arXiv preprint arXiv:1801.02610 2018.
[18) Surgan Jandial, Puneet Mangla, Sakshi Varshney, and Vineeth Balasubramanian. AdvGAN++: Harnessing Latent Layers for Adversary Generation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision Workshops, pages 0-0, 2019.
[19] Vikash Sehwag, Mung Chiang, and Prateek Mittal. SSD: A Unified Framework for Self- Supervised Outlier Detection. In ICLR, 2021.
[20] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 Dataset. http: // www.
es. toronto. edu/ kriz/ cifar. html 2014.
[21] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
[22] Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single-layer Networks in Unsupervised Feature Learning. AISTATS, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[23] Alexander Strehl and Joydeep Ghosh. Cluster Ensembles Knowledge Reuse Framework for Combining Multiple Partitions. Journal of Machine Learning Research (JMLR), 3(Dc 617, 2002.
[24] Peihao Huang, Yan Huang, Wei Wang, and Liang Wang. Deep Embedding Network for Clustering. In 2014 22nd International Conference on Pattern Recognition, pages 1532-1537. IEEE, 2014.
[25] Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian Reid. Deep Subspace Clustering Networks. Advances in Neural Information Processing Systems, 30, 2017.
[26] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering. arXiv preprint drXiv:1611.05148, 2016.
11

---

[27] Nat Dilokthanakul, Pedro AM Mediano, Marta Garnel,, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders. arXiv preprini arXiv: /6/ I. 1.02648, 2016.
[28] Warith Harchaoui, Pierre-Alexandre Mattei, and Charles Bouveyron. Deep Adversarial Gaussian Mixture Auto-Encoder for Clustering. Technical report, 2017.
[29] Jost Tobias Springenberg. Unsupervised and Semi-Supervised Learning with Categorical Generative Adversarial Networks. arXiv preprint urXiv:1511.06390 2015.
[30] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, llya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Ad- versarial Nets. Advances Neural Information Processing Systems, 29, 2016.
[31] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive Clustering. In AAAI, 2021.
[32] Battista Biggio, Ignazio Pillai, Samuel Rota Bulò, Davide Ariu, Marcello Pelillo, and Fabio Roli. Is Data Clustering in Adversarial Settings Secure? In Proceedings of the 2013 ACM workshop on Artificial Intelligence and Security, pages 87-98, 2013.
[33] Battista Biggio, Samuel Rota Bulò, Ignazio Pillai, Michele Mura, Eyasu Zemene Mequanint, Marcello Pelillo, and Fabio Roli. Poisoning Complete-linkes linkage Hierarchical Clustering. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 42-52. Springer, 2014.
[34] Jonathan Crussell and Philip Kegelmeyer. Attacking DBSCAN for Fun and Profit. In Pro- ceedings of the 2015 SIAM International Conference on Data Mining, pages 235-243. SIAM, 2015.
[35] Yinpeng Dong, Qi-An Fu, Xiao Yang. Tianyu Pang, Hang Su, Zihao Xiao. and Jun Zhu. Benchmarking adversarial robustness on image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 321-331, 2020.
[36] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-bor adversarial attacks with limited queries and information. In International Conference on Machine Learning, pages 2137-2146. PMLR, 2018.
[37] Jonathan Uesato, Brendan O'donoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning, pages 5025-5034. PMLR, 2018.
[38] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. arXiv preprint
arXiv:1706.0608. 2017.
[39] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial Training for Free! Advances in Neural Information Processing Systems, 32, 2019.
[40] Huasong Zhong, Chong Chen, Zhongming Jin, and Xian-Sheng Hua. Deep Robust Clustering by Contrastive Learning. arXiv preprint arXiv:2008. 03030, 2020.
[41] Xiangrui Zeng, Gregory Howe, and Min Xu. End-to-end Robust Joint Unsupervised Image Aligmment and Clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3854-3866, 2021.
[42] Masataka Yamaguchi, Go Irie, Takahito Kawanishi, and Kunio Kashino. Subspace Structure- aware Spectral Clustering for Robust Subspace Clustering. In Proceedings of the IEEF/CVF International Conference on Computer Vision pages 9875-9884, 2019.
[43] Benchmark: ttate-of-the-art Deep Clustering Models on CIFAR-10. https://
paperswithoom thoode. cwtcodde.comeee coom/sot//aae haa//mmaaeeeeee clustering on- cifar- 10 2021. [Online; accessed
11-May-20211
[44] Lawrence Hubert and Phipps Arabie. Comparing Partitions. Journal of Classification, 2(1):193-
218, 1985.
[45] Tao Li and Chris Ding. The Relationships Among Various Nonnegative Matrix Factorization Methods for Clustering. In Sixhh International Conference on Data Mining (ICDM'06) pages 362-371. IEEE, 2006.
12

---

[46] Huichen Li, Linyi Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang. and Bo Li. Nonlinear Projection based Gradient Estimation for Query Efficient Blackbox Attacks. In AISTATS, pages 3142-3150. PMLR, 2021.
[47]
Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. Practical Black-box Attacks on Deep Neural Networks using Efficient Query Mechanisms. In Proceedings of the European Conference on Computer Vision (ECCV), pages 154-169, 2018.
[48] Seong Joon Oh, Bernt Schiele, and Mario Fritz. Towards Reverse-engineering Black-box Neural Networks. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pages 121-144, Springer, 2019.
[49] Huiying Li, Shawn Shan, Emily Wenger. Jiayun Zhang, Haitao Zheng, and Ben Y Zhao. Blacklight: Defending Black-box Adversarial Attacks on Deep Neural Networks, arXiv preprint arXiv:2006. 14042, 2020.
[50] Jing Guo, Wei Ji. and Yun Li. Generative Networks for Adversarial Examples with Weighted Perturbations. In 2019 IEEE 14th International Conference on Intelligent Systems and Knowl- edge Engineering (ISKE), pages 778-784. IEEE, 2019.
[51] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. SCAN: Learning to Classify Images Without Labels. European Conference on Computer Vision (ECCV), pages 268-288 285. Springer, 2020.
[52] Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering. In ICLR, 2020.
[53] Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest Neighbor Matching for Deep Clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13693-13702, 2021.
[54] Yann LeCun, Corinna Cortes, and CJ Burges. MNIST Handwritten Digit Database.
http:
// yann. lecun. com/ endb/ mnist 2, 2010.
[55] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv preprint arXiv: 170.00777, 2017.
[56] Benchmark: State-of-the aww.aalyydettetioonooo -nomaly-detectionn
Anomaly Detection Models on CIFAR-10. https:// wtt/ /tta//ttaaaaaaae www.///ot//oooaaooooeeeeeeeee [Online; accessed 11-May-2021J.
paperswithcodd com
detection-on on 202I.
[57] Svante Wold, Kim Esbensen, and Paul Geladi. Principal Component Analysis. Chemometries and Intelligent Laboratory Systems, 2(1-3):37-52, 1987.
[58] Kuang-Chih Lee, J. Ho. and D.J. Kriegman. Acquiring Linear Subspaces for Face Recognition Under Variable Lighting. IEEE Transactions Pattern Analysis and Machine Intelligence, 27(5):684 698, 2005.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We have justified the claims made the introduction and abstract with a extensive results section (Section 6 and provide additional details and experiments the Appendix as well.
(b) Did you describe the limitations of your work? [Yes] We describe limitations in Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We de- scribe broader impact of our work as well as negative societal impacts in the Conclusion (Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read through them, and ensured that our paper conforms to the guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
13

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Yes, we include code, dataset, and implementation details in the Appendix D We also provide actual code along with reproducibility instructions in the supplementary material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We provide complete list of all experiment details and hyperpa- rameter values in Appendix A. for the attack on open-source models. For the defense
approaches, we provide additional experiment details in Appendix B For the attack on
Face++ we provide additional experiment details in Appendix
(c)
Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report enror bars for experiments where results will have variance (such as in the figures in Sections 6.2 and 6.3). Additionally, we provide the exact values used to create these figures in Appendix B and Appendix respectively.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide details on implementation and resources used in Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We only use existing datasets in this paper, and cite the respective authors in the main text.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "i4p8Bk5oUr", "reviewer": "Reviewer_c5rd", "summary": "This paper investigates the blackbox adversarial attacks against deep clustering models. It first proposes a blackbox adversarial attack using vanilla GAN. Then it evaluates the attacks against a number of SOTA deep clustering models and real-world images. It reveals that the proposed attack can significantly reduces the performance of these models, has minimal query complexity. Moreover, to further examine the robustness, it utilizes two unsupervised defense approaches, and observes unsuccessful defenses. And it also shows a successful attack on production-level API Face++.\n", "questions": "Besides the major questions in the \"Weaknesses\" part, there could be more deserving the authors' attentions:\n\n3. The evidence of PCA might not be convincing. We assumes you perform PCA on the total datasets, including all classes. As a complicated clustering/classification task (not easily separable w.r.t. a few principle components), it is not supervising that all data mixed in PCA plot together with the adversarial ones (considering they are close to the original).\n\n4. In Fig.6, why the \"best\" plot also has an errorbar. Did it show multiple \"best\"? How did you define the \"best\"? Maybe we missed something.\n\n5. We failed to see the necessity or the supportiveness of the application of anomaly detection as a defense approach. Considering the adversarial attack samples are \"close\" to the original, it may be foreseen that they fail to be observed as anomalies.\n\n6. The query complexity of the attack (Tbl.2) may not be a fair comparison. These attack mechanisms are different and the attacks are imposed at different stages of learning process. Not sure whether this comparison could tell something.\n", "limitations": "We do not see any potential negative societal impact of this paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper, as the authors claimed, proposed the first blackbox adversarial attack against deep clustering models. And it performed multiple experiments to show its success on various SOTA models, real-world datasets, query complexity, transferability, resistance to defenses, and production-level API Face++.\n\nWeaknesses:\n\n1. The blackbox adversarial attacks seem a straightforward application of vanilla GAN to this problem. Although the authors mentioned the challenges in building GAN-based blackbox attacks in Sec.3, we did not see any novelty or smart operations in Sec.4.1 to build up blackbox attacks. And the most challenging part of the direct formulation of attack objective has been bypassed by practically choosing the indirect, and hence the ratio of successful attacks among total raises concerns. \n\n2. The norm of adversarial noise, epsilon, may not be an appropriately interpretable one in experiments (Fig.3), since its reasonable threshold can vary significantly for different datasets. Obviously a large enough epsilon can always lead to a \"successful\" attack, while it may not be a meaningful one (it should be reasonably small such that this attack does not completely overwrite datasets). One has to formulate an index that quantifies the intensity of attacking noises over data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g-B3jkpPsCW", "reviewer": "Reviewer_sgLu", "summary": "This paper presents a new black-box attack against deep clustering models. Specifically, the adversarial perturbation is computed by a generative model. It penetrates existing defense methods for clustering, and manifests transferrability which further make the proposed method practical. To further demonstrate the effectiveness of the proposed method, experiments are also conducted on a commercial API named Face++, apart from commonly used evaluation datasets in previous related works.", "questions": "My questions are written in weaknesses 4, 5, 6, and 8.", "limitations": "Limitations are not discussed in detail -- only several words in section 7.\nPotential societal impact is discussed in the last paragraph. I agree that in order to be able to defend, we first have to know how to attack.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. The manuscript focuses on run-time adversarial attack on deep clustering model, which is a less-explored area.\n\n2. The proposed attack method is simple and intuitive. Experiments demonstrate its effectiveness.\n\n# Weaknesses\n\n1. [minor issue: paper organization] The introductory part of the manuscript can be improved. Since the proposed attack and defense methods are all based on deep learning, it is no longer necessary to introduce the clustering task starting from traditional methods like k-means. Instead, it is suggested to briefly mention (1) how a deep learning-based clustering method works, (2) how does attack against such method work, and (3) the potential risk induced by the attack to practical applications. These will make it more worthwhile for this topic to draw attention from the community (line 40).\n\n2. [minor issue: unclear figure] In figure 1, it is unclear which picture correspond to which model mentioned on line 79 (SPICE, RUC).\n\n3. [minor issue: potentially misleading] On line 35, it is pointed out that “zero-th order optimization is intractable for deep models.” in foot note. However, zero-th order optimization is a common approach for black-box attack against supervised deep classification models. See survey paper “Benchmarking Adversarial Robustness on Image Classification, CVPR2020” for examples, including ZOO, NES, as well as SPSA. In this sense, the word “intractable” is ambiguous, and may possibly convey wrong information to the reader.\n\n4. [insufficient citations] This paper focuses on attack and defense problem on a specific domain other than classification. In section 2, the author should cite and discuss the main-stream attack defense works on the classification problem, and discuss how and why these ideas cannot be adapted for clustering.\n\n5. [misleading contents] In the field of classification, “black-box attack” usually refers to run-time attacks against a trained and frozen model. See “Benchmarking Adversarial Robustness on Image Classification, CVPR2020” for examples. The attacks happend during training-time are called Backdoor attack or Poisoning attack. So please avoid using “adversarial attacks against clustering” as paragraph title on line 90. To be less ambiguous, it should be “Training-time attacks against ...”.\n\n6. [important: relationship with classification attack] Based on the description on line 116. A frozen trained clustering model is a classification model to assign cluster labels from softmax probabilities. In this way the existing black-box (run-time) attacks should be compatible in the setting of this paper. However, comparison and discussion are missing. The aforementioned CVPR2020 survey paper can be used as a referece regarding this. Existing (black-box, run-time) classification attack methods can be used to attack the cluster assignment, e.g. based on the softmax scores. To be clear, score-based attacks like NES and SPSA in the mentioned CVPR2020 paper can do this. In this sense, the claim on line 138 is wrong. Score-based attack like NES and SPSA can optimize M (softmax scores) using estimated gradients even if the optimization target is completely indifferentiable.\n\n7. [minor issue: unconvincing claim] On line 210, the ALRDC approach makes the latent space robust to perturbations. A robust latent space is already very important for adversarial robustness, and it not clear why this restricts its use. That said, evidence provided on line 212 indeed makes comparison with ALRDC unnecessary.\n\n8. [confusing figure] Figure 2 shows an irregular pattern after attack. According to equation on line 148, the attack is untargeted, which means the objective does not care which would be the result class for misclassification. In this sense, shouldn't the confusion matrix be relative random instead of being concentrated on the “cat” or “horse” class?\n\n9. [misleading title] The title mentions “defense”. But the paper actually only involves attack and evaluated this attack against existing methods for defense. The paper does not propose any new possibility for defense.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aCOPNMCshEq", "reviewer": "Reviewer_88S9", "summary": "This paper proposes the first black-box test-time adversarial attack against deep clustering models. This work helps bridge the adversarial analysis from traditional clustering approaches (explored before) to deep learning-based clustering models, where only white-box supervised learning based attacks have been explored. The attack uses Generative Adversarial Networks (GAN) with just score-based query access to the deep clustering model, with a more clearly defined threat model. The authors evaluate this attack on SOTA deep clustering models, in a transferability study, and on the Face++ ML-as-a-Service (MLaaS) system. The authors also demonstrate that existing defense approaches fail to stop the attack.", "questions": "The paper is well-written and explores an interesting problem (the adversarial robustness of deep clustering techniques along with some mitigation strategies). Overall, I think this is a good submission, and am encouraged to see work on black-box attacks on real MLaaS systems.\n\nOne aspect that could be improved is in making further distinctions between the authors' attack and AdvGAN, as it seems that there are many technical similarities between AdvGAN and the GAN formulation in this work. Although it is adapted somewhat to the clustering problem, the overall formulation with an adversarial, hinge, and GAN loss is very similar to AdvGAN, limiting its technical novelty.\n\nWhile generally well-written, there remain some things that could be further clarified in the paper. How is the surrogate to Face++ trained? Also, how are the epsilon bounds set for the table? It seems arbitrary given that the numbers in the Appendix are not consistent across datasets and it is unclear why they are different, so it seems some justification would help. Also in this case then, for the transferability studies, are the epsilons still unequal?\n\nDo you have any insight into: 1) why there seems to be common specific patterns in the noise and 2) why most attacks are clustered in the same class?\n\nFinally, I think more examples of specific real-world threat vectors could be included to better: 1) scope the societal impact of the work and 2) better motivate the need for robust deep clustering techniques.\n\nMinor comments:\n- Could clarify a bit what the specific threat model contribution is when brought up in the Intro (line 46)\n- Intro: could cite some specific numbers on percentage of performance changes with the attack\n- Could further distinguish between when a \"score-based\" black-box model is used vs. a \"decision-based\" black-box model\n", "limitations": "The authors have provided clear descriptions on the level of access needed to data, models, etc. to pull off the attack, and several mitigation approaches are mentioned and studied. The authors also discussed their hope that this work leads to successful development of defenses. Specific examples of real-world motivations and consequences of attacks on deep clustering algorithms could be further discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Attacks a real-world MLaaS\n- Evaluates against potential mitigation strategies\n- Strong attack results\n\nWeaknesses:\n- Limited technical novelty\n- Parts of the problem have been previously explored (e.g., traditional clustering, or white-box attacks on deep clustering), although the combination of factors is new\n- Ambiguous epsilon bounds\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DFTH9hF98FB", "reviewer": "Reviewer_hxny", "summary": "This paper proposes an effective attack method against deep clustering models. Specifically, the threat model and attack objective are newly defined, reducing the attack to an optimization problem. Then, by combining a common GAN training loss with the attack objective, the authors design a GAN-based blackbox attack so that the resulting GAN generates adversarial perturbations for a given input image.\n\nThe power of the attack–effectiveness, query complexity, and transferability–is demonstrated from thorough experiments. Also, the authors tested the performance of their attack against two standard defense methods (robust deep clustering and deep learning based anomaly detection), showing successful attack results against them. Finally, they tested a blackbox attack against an existing face clustering service. Since the actual cluster membership is unknown in this case, the authors trained an open-source surrogate model and attacked via the transferability of the attack.\n", "questions": "1. While the paper demonstrated impressive performance against state-of-the-art deep clustering models, it would be interesting to attack some earlier deep clustering models. Since earlier models decompose the high-dimensional data to a low-dimensional representation, we can observe what happens in the low-dimensional space. (Those small perturbations in high dimensional may result in considerable differences in the low dimensional data.)\n2. A follow-up suggestion from Question 1 is to perform anomaly detection over the low-dimensional representations of adversarial samples. To explain, prepare an earlier version of the deep clustering model as an embedding method. (Even though earlier versions have low performance in clustering tasks, it is okay because we will use it only to create low-dimensional representations) Then, create adversarial examples against a state-of-the-art model and embed those adversarial examples in low-dimensional space. Using any anomaly detection method, check whether the embedded representations are outliers or not.\n", "limitations": "The authors discussed limitations and potential negative societal impact in the paper appropriately.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: To the best of my knowledge, the paper contains novel ideas.\n\n[[Strength]]\n1. To the best of my knowledge, this is the first work about adversarial attacks against the deep clustering model.\n\nQuality: \n\n[[Strength]]\n1. The authors tested the attack against numerous clustering models and performance metrics. To explain, for the attack performance test, the authors prepare six different state-of-the-art models and three different clustering performance measures.\n2. Moreover, the authors also explore various aspects of the attack–query complexity and transferability–to better evaluate the attack’s value.\n3. The power of the attack is evaluated carefully against existing defense strategies.\n\n[[Weakness]]\n1. Visualization of the PCA patterns between adversarial examples and original samples may not be an effective way to analyze the reason.  A better visualization should also show the difference in the PCA components between detected adversarial examples, undetected adversarial examples, and benign original samples. (The last two sets of samples should show similar patterns, while the first set should show visually different patterns.)\n\nClarity: The paper writing is clear, and there is no issue with the clarity of the paper.\n\nSignificance: \n\n[[Strength]]\n1. The power of attack looks to be very impressive. The attack significantly reduces the clustering performance for all 18 combinations (six different models, three different metrics).\n2. The proposed attack seems to be effective against existing defenses. The experimental results may imply the need for a new defense method to ensure the robustness of deep clustering algorithms.\n3. The authors experiment with the attack transferability and test it against an existing face clustering service (via an open-source surrogate model). This result shows an urgent threat of the attack in a practical setting.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept